YARN Application Master unable to connect to Resource Manager

Question

I have a 4 node cluster (1 Namenode/Resource Manager 3 datanodes/node managers)

I am trying to run a simple tez example orderedWordCount

hadoop jar          


        

Answer 1

There is a problem in the Hadoop2 code with configuring the yarn.resourcemanager.scheduler.address e.g.:

    yarn.resourcemanager.scheduler.address
    qadoop-nn001.apsalar.com:8030

It is currently not properly placed into the 'conf' configuration at hadoop-2.7.0/src/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/RMProxy.java

To prove the issue, we patched that file to directly inject our scheduler address. The patch below is a hack. The root cause is with the 'conf' object that needs to load the property "yarn.resourcemanager.scheduler.address".

    @Private
protected static  T createRMProxy(final Configuration configuration, final Class protocol, RMProxy instance) throws IOException {
    YarnConfiguration conf = (configuration instanceof YarnConfiguration)
        ? (YarnConfiguration) configuration
        : new YarnConfiguration(configuration);
    LOG.info("LEE: changing the conf to include yarn.resourcemanager.scheduler.address at 10.1.26.1");
    conf.set("yarn.resourcemanager.scheduler.address", "10.1.26.1");
    RetryPolicy retryPolicy = createRetryPolicy(conf);
    if (HAUtil.isHAEnabled(conf)) {
      RMFailoverProxyProvider provider =
          instance.createRMFailoverProxyProvider(conf, protocol);
      return (T) RetryProxy.create(protocol, provider, retryPolicy);
    } else {
      InetSocketAddress rmAddress = instance.getRMAddress(conf, protocol);
      LOG.info("LEE: Connecting to ResourceManager at " + rmAddress);
      T proxy = RMProxy.getProxy(conf, protocol, rmAddress);
      return (T) RetryProxy.create(protocol, proxy, retryPolicy);
    }   
} 

EDIT: we solved this problem by adding yarn-site.xml to the CLASSPATH. there is no need to modify RMProxy.java