Submitting a Hadoop job

问题

I need to constantly get the mappers' and reducers' running time. I have submitted the job as follows.

 JobClient jobclient = new JobClient(conf);
 RunningJob runjob = jobclient.submitJob(conf);          


 TaskReport [] maps = jobclient.getMapTaskReports(runjob.getID());

 long mapDuration = 0;
 for(TaskReport rpt: maps){
    mapDuration += rpt.getFinishTime() - rpt.getStartTime();
 }

However when I run the program, it seems like the job is not submitted and the mapper never starts. How can I use JobClient.runJob(conf) and still be able to get the running time?

回答1:

The submitJob() method returns control immediately to the calling program without waiting for the hadoop Job to start, much less complete. If you want to wait then use the waitForCompletion() method which returns only after the hadoop job has finished. I think you want something in between since you want to run subsequent code after the submit but before the complete.

I suggest you put your follow-on code in a loop that continues until the job is complete (Use the isComplete() method for that test) and observe the mappers and reducers as the job progresses. You probably want to put a Thread.sleep(xxx) in the loop somewhere, too.

To respond to your comment, you want to...

job.waitForCompletion();
TaskCompletionEvent event[] = job.getTaskCompletionEvents();
for (int i = 0; i < event.length(); i++) {
    System.out.println("Task "+i+" took "+event[i].getTaskRunTime()+" ms");
}    

来源：https://stackoverflow.com/questions/16778939/submitting-a-hadoop-job