How to restart HDFS on Amazon EMR

问题

I have made some changes in the settings for HDFS on an Amazon EMR cluster. I want to restart the namenode and the datanode for the changes to take effect. I am not able to find any start and stop scripts to do so on neither the namenode(master) nor the datanodes. What should be the way to restart the cluster?

回答1:

On EMR4 , run following on master host -

sudo /sbin/start hadoop-hdfs-namenode
ssh -i <key.pem> <slave-hostname1> "sudo /sbin/restart  hadoop-hdfs-datanode"
ssh -i <key.pem> <slave-hostname2> "sudo /sbin/restart  hadoop-hdfs-datanode"
ssh -i <key.pem> <slave-hostname3> "sudo /sbin/restart  hadoop-hdfs-datanode"

回答2:

You have to manually restart the cluster. This can be either performed manually or using a simple shell script.

1) Get the list of hostnames or ipaddress of all the nodes,

2) ssh into the node using the key

3) Restart the required service.

If you are good in programming, you can create a general utility that will get the list of ipaddress of all the nodes corresponding to an EMR by using the cluster id and perform the service restart in individual nodes.

Otherwise, get the hostname or ipaddress of all the nodes manually and create a script like the below one and execute from the master node

sudo service hadoop-hdfs-namenode restart
ssh -i <key.pem> <hostname1> "sudo service hadoop-hdfs-datanode restart"
ssh -i <key.pem> <hostname2> "sudo service hadoop-hdfs-datanode restart"
ssh -i <key.pem> <hostname3> "sudo service hadoop-hdfs-datanode restart"

回答3:

On EMR 5.x this is what I used:

Copy PEM file to your head node and set these values:
```
CLUSTER_ID="j-XXXXXXXXXXX"
IDENT="cluster.pem"
```

Run this:

nodes=$(aws emr list-instances \
    --cluster-id $ \
    --instance-group-types CORE \
    --instance-states RUNNING \
    --output text \
    --query "Instances[*].PublicDnsName" )

for node in nodes; do
    ssh -i $IDENT \
        -o StrictHostKeyChecking=no \
        -o UserKnownHostsFile=/dev/null \
        $node "sudo stop hadoop-hdfs-datanode; sudo start hadoop-hdfs-datanode"
done

来源：https://stackoverflow.com/questions/32207723/how-to-restart-hdfs-on-amazon-emr

标签

Hadoop

HDFS

emr