I am trying to push my Spark processed data to 3 node cluster of C*. I am pushing 200 million records to Cassandra it is failing with below err.
Below it the my spark cluster configuration
Nodes : 12
vCores Total : 112
Total memory : 1.5 TB.
Below are my spark-sumbit parameters:
$SPARK_HOME/bin/spark-submit
--master yarn
--deploy-mode cluster
--name app
--class Driver
--executor-cores 3
--executor-memory 8g
--num-executors 10
--driver-cores 2
--driver-memory 10g
--conf spark.shuffle.service.enabled=true
--conf spark.dynamicAllocation.enabled=false
--conf spark.task.maxFailures=8
--conf spark.ui.killEnabled=false
oracle2c.jar
I have made the Spark data frame partitions 30 as below
+--------------------+-------+
|SPARK_PARTITION_ID()| count|
+--------------------+-------+
| 22|6687176|
| 29|6687175|
| 8|6687176|
| 21|6687176|
| 23|6687176|
| 5|6687176|
| 9|6687176|
| 17|6687176|
| 26|6687175|
| 27|6687175|
| 4|6687176|
| 10|6687176|
| 18|6687176|
| 25|6687175|
| 1|6687176|
| 20|6687176|
| 12|6687176|
| 28|6687175|
| 24|6687176|
| 15|6687176|
| 14|6687176|
| 6|6687176|
| 19|6687176|
| 0|6687175|
| 3|6687176|
| 11|6687176|
| 2|6687176|
| 7|6687176|
| 13|6687176|
| 16|6687176|
+--------------------+-------+
But when I run the job I see only one core in executor is running , how to make all the cores take part of load distribution?
What more configuration parameters I need to add to make all the cores take the load ?
来源：https://stackoverflow.com/questions/53442130/why-only-one-core-is-taking-all-the-load-how-to-make-other-29-cores-to-take-lo