问题
I would like to call spark jar from java (to run spark process on yarn), and try to use this link code.
It looks fit in my case, but I need to pass hashmap and some java values to spakr jar. Is it able to pass java object to spark jar?
And is java side able to know how mush spark jar process or is it done? if so, how?
回答1:
I think you misunderstood the content given in data-algorithms
There are 2 ways to submit job
1) Spark-submit like below example from shell script
cat run_secondarysorting.sh
#!/bin/bash
export JAVA_HOME=/usr/java/jdk7
export SPARK_HOME=/home/hadoop/spark-1.1.0
export SPARK_MASTER=spark://myserver100:7077
BOOK_HOME=/home/mp/data-algorithms-book
APP_JAR=$BOOK_HOME/dist/data_algorithms_book.jar
INPUT=/home/hadoop/testspark/timeseries.txt
# Run on a Spark standalone cluster
prog=org.dataalgorithms.chap01.spark.SparkSecondarySort
$SPARK_HOME/bin/spark-submit \
--class $prog \
--master $SPARK_MASTER \
--executor-memory 2G \
--total-executor-cores 20 \
$APP_JAR
2) From Yarn Client which was described in the link.
Usage of Yarn Client is
want to submit Spark jobs from Java code (such as Java servlets or other Java code such as REST servers).
When you are calling this yarn client... then you need to call as method in your rest-service or servlet etc... (i.e. through web) in which you can also pass parameters like HashMap or any java object kind...
For demo purpose he has written standalone client (with public static void main)
Hope you understood..
来源：https://stackoverflow.com/questions/41113712/spark-call-spark-jar-from-java-with-arguments