MapReduce job hangs, waiting for AM container to be allocated

Question

I tried to run simple word count as MapReduce job. Everything works fine when run locally (all work done on Name Node). But, when I try to run it on a cluster using YARN (ad

Answer 1

I feel, you are getting your memory settings wrong.

To understand the tuning of YARN configuration, I found this to be a very good source: http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html

I followed the instructions given in this blog and was able to get my jobs running. You should alter your settings proportional to the physical memory you have on your nodes.

Key things to remember is:

• Values of mapreduce.map.memory.mb and mapreduce.reduce.memory.mb should be at least yarn.scheduler.minimum-allocation-mb
• Values of mapreduce.map.java.opts and mapreduce.reduce.java.opts should be around "0.8 times the value of" corresponding mapreduce.map.memory.mb and mapreduce.reduce.memory.mb configurations. (In my case it is 983 MB ~ (0.8 * 1228 MB))
• Similarly, value of yarn.app.mapreduce.am.command-opts should be "0.8 times the value of" yarn.app.mapreduce.am.resource.mb

Following are the settings I use and they work perfectly for me:

yarn-site.xml:

 
    yarn.scheduler.minimum-allocation-mb 
    1228

 
    yarn.scheduler.maximum-allocation-mb 
    9830

 
    yarn.nodemanager.resource.memory-mb 
    9830

mapred-site.xml

  
    yarn.app.mapreduce.am.resource.mb  
    1228

 
    yarn.app.mapreduce.am.command-opts 
    -Xmx983m


    mapreduce.map.memory.mb
    1228


    mapreduce.reduce.memory.mb
    1228


    mapreduce.map.java.opts
    -Xmx983m


    mapreduce.reduce.java.opts
    -Xmx983m

You can also refer to the answer here: Yarn container understanding and tuning

You can add vCore settings, if you want your container allocation to take into account CPU also. But, for this to work, you need to use CapacityScheduler with DominantResourceCalculator. See the discussion about this here: How are containers created based on vcores and memory in MapReduce2?