Spark : check your cluster UI to ensure that workers are registered

Question

I have a simple program in Spark:

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.S         


        

Answer 1

You can check your cluster's work node cores: your application can't exceed that. For example, you have two work node. And per work node you have 4 cores. Then you have 2 applications to run. So you can give every application 4 cores to run the job.

You can set like this in the code:

SparkConf sparkConf = new SparkConf().setAppName("JianSheJieDuan")
                          .set("spark.cores.max", "4");

It works for me.

Answer 2

There are also some causes of this same error message other than those posted here.

For a spark-on-mesos cluster, make sure you have java8 or newer java version on mesos slaves.

For spark standalone, make sure you have java8 (or newer) on the workers.

Answer 3

Finally figured out what the answer is.

When deploying a spark program on a YARN cluster, the master URL is just yarn.

So in the program, the spark context should just looks like:

val conf = new SparkConf().setAppName("SimpleApp")

Then this eclipse project should be built using Maven and the generated jar should be deployed on the cluster by copying it to the cluster and then running the following command

spark-submit --master yarn --class "SimpleApp" Recommender_2-0.0.1-SNAPSHOT.jar

This means that running from eclipse directly would not work.

Answer 4

I have done configuration and performance tuning for many spark clusters and this is a very common/normal message to see when you are first prepping/configuring a cluster to handle your workloads.

This is unequivocally due to insufficient resources to have the job launched. The job is requesting one of:

• more memory per worker than allocated to it (1GB)
• more CPU's than available on the cluster

Answer 5

You don't have any workers to execute the job. There are no available cores for the job to execute and that's the reason the job's state is still in 'Waiting'.

If you have no workers registered with Cloudera how will the jobs execute?