Spark Launcher waiting for job completion infinitely
I am trying to submit a JAR with Spark job into the YARN cluster from Java code. I am using SparkLauncher to submit SparkPi example: Process spark = new SparkLauncher() .setAppResource("C:\\spark-1.4.1-bin-hadoop2.6\\lib\\spark-examples-1.4.1-hadoop2.6.0.jar") .setMainClass("org.apache.spark.examples.SparkPi") .setMaster("yarn-cluster") .launch(); System.out.println("Waiting for finish..."); int exitCode = spark.waitFor(); System.out.println("Finished! Exit code:" + exitCode); There are two problems: While submitting in "yarn-cluster" mode, the application is sucessfully submitted to YARN and