How to increase the number of containers in nodemanager in YARN

问题

A node in my YARN cluster has 64GB memory and 24 cores. I set the following properties in the yarn-site.xml:

<property>
  <name>yarn.nodemanager.resource.memory-mb</name>
  <value>32768</value>
</property>
<property>
  <name>yarn.nodemanager.resource.cpu-vcores</name>
  <value>16</value>
</property>

But I found still the nodemanager in the node has only 7 containers. What are other properties I need to set?

回答1:

You need to tell YARN how to break down the memory to containers so for instance if you set the memory per container to 2GB will give you 16 containers

<name>yarn.scheduler.minimum-allocation-mb</name>
<value>2048</value>

回答2:

try something like:

  <property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>      
  </property>

  <property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>2048</value>      
  </property>

来源：https://stackoverflow.com/questions/20021566/how-to-increase-the-number-of-containers-in-nodemanager-in-yarn