How to change memory per node for apache spark worker

Question

I am configuring an Apache Spark cluster.

When I run the cluster with 1 master and 3 slaves, I see this on the master monitor page:

Memory
2.0 GB (51         


        

Answer 1

According to Spark documentation you can change the Memory per Node with command line argument --executor-memory while submitting your application. E.g.

./bin/spark-submit \
  --class org.apache.spark.examples.SparkPi \
  --master spark://master.node:7077 \
  --executor-memory 8G \
  --total-executor-cores 100 \
  /path/to/examples.jar \
  1000

I've tested and it works.