Spark - Error “A master URL must be set in your configuration” when submitting an app

Question

I have an Spark app which runs with no problem in local mode,but have some problems when submitting to the Spark cluster.

The error msg are as follows:

Answer 1

How does spark context in your application pick the value for spark master?

• You either provide it explcitly withing SparkConf while creating SC.
• Or it picks from the System.getProperties (where SparkSubmit earlier put it after reading your --master argument).

Now, SparkSubmit runs on the driver -- which in your case is the machine from where you're executing the spark-submit script. And this is probably working as expected for you too.

However, from the information you've posted it looks like you are creating a spark context in the code that is sent to the executor -- and given that there is no spark.master system property available there, it fails. (And you shouldn't really be doing so, if this is the case.)

Can you please post the GroupEvolutionES code (specifically where you're creating SparkContext(s)).