I'm using the spark-submit command I have for the log4j properties to invoke a Spark-submit like this:
/opt/spark-1.6.2-bin-hadoop2.6/bin/spark-submit \ --driver-java-options \ "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties\ --class Test testing.jar How do I do --driver-java-options, to submit a job via curl (Apache Spark's Hidden REST API)?
I tried this:
curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{ "action" : "CreateSubmissionRequest", "appArgs" : [ "" ], "appResource" : "hdfs://host-ip:9000/test/testing.jar", "clientSparkVersion" : "1.6.2", "environmentVariables" : { "SPARK_ENV_LOADED" : "1" }, "mainClass" : "Test", "spark.driver.extraJavaOptions" : "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties", "sparkProperties" : { "spark.jars" : "hdfs://host-ip:9000/test/testing.jar", "spark.app.name" : "Test", "spark.eventLog.enabled": "true", "spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events", "spark.submit.deployMode" : "cluster", "spark.master" : "spark://host-ip:7077" } }' Job submitted successfully and response was given, but with one uknownField:
{ "action" : "CreateSubmissionResponse", "message" : "Driver successfully submitted as driver-20160810210057-0091", "serverSparkVersion" : "1.6.2", "submissionId" : "driver-20160810210057-0091", "success" : true, "unknownFields" : [ "spark.driver.extraJavaOptions" ] } "unknownFields" : [ "spark.driver.extraJavaOptions" ]
I have also tried driverExtraJavaOptions as follows:
curl -X POST http://host-ip:6066/v1/submissions/create --header "Content-Type:application/json;charset=UTF-8" --data '{ "action" : "CreateSubmissionRequest", "appArgs" : [ "" ], "appResource" : "hdfs://host-ip:9000/test/testing.jar", "clientSparkVersion" : "1.6.2", "environmentVariables" : { "SPARK_ENV_LOADED" : "1" }, "mainClass" : "Test", "driverExtraJavaOptions" : "-Dlog4j.configuration=file:/home/test_api/log4j-driver.properties", "sparkProperties" : { "spark.jars" : "hdfs://host-ip:9000/test/testing.jar", "spark.app.name" : "Test", "spark.eventLog.enabled": "true", "spark.eventLog.dir": "hdfs://host-ip:9000/test/spark-events", "spark.submit.deployMode" : "cluster", "spark.master" : "spark://host-ip:7077" } }' But got a similar response:
{ "action" : "CreateSubmissionResponse", "message" : "Driver successfully submitted as driver-20160810211432-0094", "serverSparkVersion" : "1.6.2", "submissionId" : "driver-20160810211432-0094", "success" : true, "unknownFields" : [ "driverExtraJavaOptions" ] } Why is this?
I looked at spark-submit.scala and referenced the Spark REST API