How to set OutputCommitter config?

问题

I have a mapreduce code that submits job using JobClient . I keep getting this null pointer error stack

12/12/10 12:42:44 INFO mapred.LocalJobRunner: OutputCommitter set in config null    
Exception in thread "main" java.lang.NullPointerException
        at org.apache.hadoop.mapred.JobClient$NetworkedJob.<init>(JobClient.java:226)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:924)
        at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:844)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
        at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:844)
        at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:818)
        at FTPIF.run(FTPIF.java:193)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
        at FTPIF.main(FTPIF.java:273)

The piece of code that gets me this error is

JobClient j = new JobClient();
        j.init(conf);
        RunningJob check = j.submitJob(conf);

Any ideas ?

回答1:

I have this problem as well. But using j.runJob(conf); works well. Also you can use the static function JobClient.runJob(conf);. I do not know why, then I read the source of JobClient and find the implementation of the static function runJob:

public static RunningJob runJob(JobConf job) throws IOException {
    JobClient jc = new JobClient(job);
    RunningJob rj = jc.submitJob(job);
    try {
        if (!jc.monitorAndPrintJob(job, rj)) {
            throw new IOException("Job failed!");
        }
    } catch (InterruptedException ie) {
        Thread.currentThread().interrupt();
    }
    return rj;
}

So I use the submitJob function in this way:

JobClient jcli = new JobClient(jconf);
RunningJob rJob = jcli.submitJob(jconf);
while (true){
    Thread.sleep(5000);
    System.out.println(rJob);
    if (rJob.isComplete())
        break;
}

I works!! Maybe this is also useful to you.

回答2:

I never figured out why I hit on the null pointer . Setting j.setConf(conf) also did not work . So I used runJob instead to get a handle on the running job .

RunningJob job= JobClient.runJob(conf);

来源：https://stackoverflow.com/questions/13796625/how-to-set-outputcommitter-config

标签

Hadoop

MapReduce