Spark : check your cluster UI to ensure that workers are registered

Question

I have a simple program in Spark:

/* SimpleApp.scala */
import org.apache.spark.SparkContext
import org.apache.spark.SparkContext._
import org.apache.spark.S         


        

Answer 1

I have done configuration and performance tuning for many spark clusters and this is a very common/normal message to see when you are first prepping/configuring a cluster to handle your workloads.

This is unequivocally due to insufficient resources to have the job launched. The job is requesting one of:

• more memory per worker than allocated to it (1GB)
• more CPU's than available on the cluster