(Hadoop) MapReduce - Chain jobs - JobControl doesn't stop

Question

I need to chain two MapReduce jobs. I used JobControl to set job2 as dependent of job1. It works, output files are created!! But it doesn\'t stop! In the shell it remains in

Answer 1

I essentially did what Pietro alluded to above.

public class JobRunner implements Runnable {
  private JobControl control;

  public JobRunner(JobControl _control) {
    this.control = _control;
  }

  public void run() {
    this.control.run();
  }
}

and in my map/reduce class I have:

public void handleRun(JobControl control) throws InterruptedException {
    JobRunner runner = new JobRunner(control);
    Thread t = new Thread(runner);
    t.start();

    while (!control.allFinished()) {
        System.out.println("Still running...");
        Thread.sleep(5000);
    }
}

in which I just pass the jobControl object.

Answer 2

The JobControl object itself is Runnable, so you can just use it like this:

new Thread(myJobControlInstance).start()

Answer 3

try this:

    Thread jcThread = new Thread(jobControl);
    jcThread.start();
    System.out.println("循环判断jobControl运行状态 >>>>>>>>>>>>>>>>");
    while (true) {
        if (jobControl.allFinished()) {
        System.out.println("====>> jobControl.allFinished=" + jobControl.getSuccessfulJobList());
        jobControl.stop();
        // 如果不加 break 或者 return，程序会一直循环
        break;
    }

    if (jobControl.getFailedJobList().size() > 0) {
        succ = 0;
        System.out.println("====>> jobControl.getFailedJobList=" + jobControl.getFailedJobList());
        jobControl.stop();

        // 如果不加 break 或者 return，程序会一直循环
        break;
    }
}

Answer 4

Just a tweak to the code snippet what sinemetu1 had shared..

You can drop call to the JobRunner as JobControl by itself implements Runnable

        Thread thread = new Thread(jobControl);
        thread.start();

        while (!jobControl.allFinished()) {
            System.out.println("Still running...");
            Thread.sleep(5000);
        }

I also stumbled upon this link where the user confirms that JobControl can be run ONLY with new thread. https://www.mail-archive.com/common-user@hadoop.apache.org/msg00556.html