问题
In spark job. I am using if file not found the system.exit(0). It should gracefully complete the job. Locally It is successfully completed. But when I am running on EMR. Step is failing.
回答1:
EMR uses YARN for cluster management and launching Spark applications. So when you're running a Spark app with --deploy-mode: cluster in EMR, the Spark application code is not running in a JVM on its own, but is rather executed by the ApplicationMaster class.
Browsing through the ApplicationMaster code can explain what happens when you try to execute System.exit(). The user application is launched in startUserApplication, and then the finish method is called after the user application returns. However, when you call System.exit(0), what is executed instead is a shutdown hook which sees that your code hasn't finished successfully, and marks it as an EXIT_EARLY failure. It also has this useful comment:
// The default state of ApplicationMaster is failed if it is invoked by shut down hook.
// This behavior is different compared to 1.x version.
// If user application is exited ahead of time by calling System.exit(N), here mark
// this application as failed with EXIT_EARLY. For a good shutdown, user shouldn't call
// System.exit(0) to terminate the application.
来源：https://stackoverflow.com/questions/39580449/spark-jobs-running-on-emr-cluster-system-exit0-used-to-gracefully-completion