Spark Mesos Dispatcher

Question

My team is deploying a new Big Data architecture on Amazon Cloud. We have Mesos up and running Spark jobs.

We are submitting Spark jobs (i.e.: jars) from a bastion host inside the same cluster. Doing so, however, the bastion host is the driver program and this is called the client mode (if I understood correctly).

We would like to try the cluster mode, but we don't understand where to start the dispatcher process.

The documentation says to start it in the cluster, but I'm confused since our masters don't have Spark installed and we use Zookeeper for master election. Starting it on a slave node is not a vailable option, since slave can fail and we don't want to expose a slave ip or public DNS to the bastion host.

Is it correct to start the dispatcher on the bastion host?

Thank you very much

Answer 1

Documentation is not very detailed. However, we are quite happy with what we discovered: according to the documentation, cluster mode is not supported for Mesos clusters (and for Python applications).

However, we started the dispatcher using --master mesos://zk://...

For submitting applications, you need the following:

spark-submit --deploy-mode cluster <other options> --master mesos://<dispatcher_ip>:7077 <ClassName> <jar>

If you run this command from a bastion machine, it won't work, because the Mesos master will look for the submitable jar in the same path as the bastion. We ended exposing the file as a downloadable URL.

Hope this helps

Answer 2

I haven't used cluster mode in Mesos and the cluster mode description is not very detailed. There isn't even a --help option on the script, like there should be, IMHO. However, if you don't pass the --master argument, it errors out with a help message and it turns out there is a --zk option for specifying the Zookeeper URL.

What might work is to launch this script on the bastion itself with the appropriate --master and --zk options. Would that work for you?

Answer 3

You could use a docker image with spark and your application.jar instead of uploading the jar to s3. I didn't try yet, but I think it should work. The environment variable is SPARK_DIST_CLASSPATH in spark-env.sh. I use spark distribution compiled without hadoop with apache hadoop 2.7.1

export SPARK_DIST_CLASSPATH=$(/opt/hadoop/bin/hadoop classpath):/opt/hadoop/share/hadoop/tools/lib/*:/opt/application.jar