yarn is not honouring yarn.nodemanager.resource.cpu-vcores

Question

I am using Hadoop-2.4.0 and my system configs are 24 cores, 96 GB RAM.

I am using following configs

mapreduce.map.cpu.vcores=1
yarn.node         


        

Answer 1

YARN is running more containers than allocated cores because by default DefaultResourceCalculator is used. It considers only memory.

public int computeAvailableContainers(Resource available, Resource required) {
// Only consider memory
return available.getMemory() / required.getMemory();
  }

Use DominantResourceCalculator, It uses both cpu and memory.

Set below config in capacity-scheduler.xml

yarn.scheduler.capacity.resource-calculator=org.apache.hadoop.yarn.util.resource.DominantResourceCalculator

More about DominantResourceCalculator