Spark mesos cluster mode is slower than local mode

问题

I submit the same jar to run by using both local mode and mesos cluster mode. And found for some exactly same stages, local mode only takes several milliseconds to finish however cluster mode will take seconds!

listed is one example: stage 659

local mode: 659 Streaming job from [output operation 1, batch time 17:45:50] map at KafkaHelper.scala:35 +details 2016/03/22 17:46:31 11 ms

mesos cluster mode: 659 Streaming job from [output operation 1, batch time 18:01:20] map at KafkaHelper.scala:35 +details 2016/03/22 18:09:33 3 s

And I found from spark UI that mesos cluster mode will consistently take 4 seconds to finish the foreachRDD jobs, why is that? Any submit commands options can help with this?

Bunch of thanks in advance!

回答1:

That behavior depends on multiple factors. You don't specify what kind of job you run in which cluster mode, and with which settings. If Spark is not installed on the Slaves, you'll see an overhead because the distribution needs to be downloaded etc.

Furthermore, the jars you're using need to be distributed to the executors, which can take some time for the startup as well.

As said, this all depends on how you run Spark on Mesos.

See