How to restart yarn on AWS EMR

Question

I am using Hadoop 2.6.0 (emr-4.2.0 image). I have made some changes in yarn-site.xml and want to restart yarn to bring the changes into effect.

Answer 1

For those who are gonna come from Google

In order to restart a service in EMR, perform the following actions:

• Find the name of the service by running the following command:

  initctl list
  

For example, the YARN Resource Manager service is named hadoop-yarn-resourcemanager.

• Stop the service by running the following command:

  sudo stop hadoop-yarn-resourcemanager
  

• Wait a few seconds, then start the service by running the following command:

  sudo start hadoop-yarn-resourcemanager
  

Note: Stop/start is required; do not use the restart command.

• Verify that the process is running by running the following command:

  sudo status hadoop-yarn-resourcemanager
  

Check for the process using ps, and then check the log file for any errors in the log directory /var/log/.

Source : https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/

Answer 2

Try restarting this service as well:

hadoop-yarn-nodemanager

Answer 3

Edit (10/26/2017): A more detailed Knowledge Center article on how to do this has been published here by AWS officially - https://aws.amazon.com/premiumsupport/knowledge-center/restart-service-emr/.

You can ssh into the master node of your EMR cluster and run -

"sudo /sbin/stop hadoop-yarn-resourcemanager"
"sudo /sbin/start hadoop-yarn-resourcemanager"

commands to restart the Yarn resource manager. EMR AMI 4.x.x uses upstart - /sbin/{start,stop,restart} are all symlinks to /sbin/initctl, which is part of upstart. See the initctl man page for more information.

Alternatively, you can follow the instructions here to propagate your changes to yarn-site.xml - yarn-change-configuration-on-yarn-site-xml

Answer 4

If what you want to do is to enable log-aggregation, it is actually easier to create the cluster with log-aggregation already enabled, as described in the documentation:

http://docs.aws.amazon.com/ElasticMapReduce/latest/ManagementGuide/emr-plan-debugging.html

(It is actually enabled by default if you are using emr-4.3.0).