问题
I'm using Spark in a YARN cluster (HDP 2.4) with the following settings:
- 1 Masternode
- 64 GB RAM (50 GB usable)
- 24 cores (19 cores usable)
- 5 Slavenodes
- 64 GB RAM (50 GB usable) each
- 24 cores (19 cores usable) each
- YARN settings
- memory of all containers (of one host): 50 GB
- minimum container size = 2 GB
- maximum container size = 50 GB
- vcores = 19
- minimum #vcores/container = 1
- maximum #vcores/container = 19
When I run my spark application with the command spark-submit --num-executors 30 --executor-cores 3 --executor-memory 7g --driver-cores 1 --driver-memory 1800m ... YARN creates 31 containers (one for each executor process + one driver process) with the following settings:
- Correct: Master container with 1 core & ~1800 MB RAM
- Correct: 30 slave containers with ~7 GB RAM each
- BUT INCORRECT: each slave container only runs with 1 core instead of 3, according to the YARN ResourceManager UI (it shows only 31 of 95 in use, instead of 91 = 30 * 3 + 1), see screenshot below
My question here: Why does the spark-submit parameter --executor-cores 3 have no effect?
回答1:
Ok, seems to be the same issue as discussed here: yarn is not honouring yarn.nodemanager.resource.cpu-vcores The solution also worked for me.
回答2:
When using Ambari:
- Select YARN Queue Manager from top right corner
- Scheduler section
- Change "Calculator" drop-down to = "Dominant Resource"
回答3:
The UI could be misleading. When vcore enforcement is disabled in YARN, it will just assume one core per container, whereas in fact, if you look at the actual CPU usage of the physical machine, it could be using all of the cores. Also check the core usage on your SparkUI, that usually gives a better understanding of the actual CPU usage.
来源：https://stackoverflow.com/questions/38368985/spark-on-yarn-too-less-vcores-used