Datanode process not running in Hadoop

Question

I set up and configured a multi-node Hadoop cluster using this tutorial.

When I type in the start-all.sh command, it shows all the processes initializing properly as

Answer 1

I have got details of the issue in the log file like below : "Invalid directory in dfs.data.dir: Incorrect permission for /home/hdfs/dnman1, expected: rwxr-xr-x, while actual: rwxrwxr-x" and from there I identified that the datanote file permission was 777 for my folder. I corrected to 755 and it started working.

Answer 2

I ran into the same issue. I have created a hdfs folder '/home/username/hdfs' with sub-directories name, data, and tmp which were referenced in config xml files of hadoop/conf.

When I started hadoop and did jps, I couldn't find datanode so I tried to manually start datanode using bin/hadoop datanode. Then I realized from error message that it has permissions issue accessing the dfs.data.dir=/home/username/hdfs/data/ which was referenced in one of the hadoop config files. All I had to do was stop hadoop, delete the contents of /home/username/hdfs/tmp/* directory and then try this command - chmod -R 755 /home/username/hdfs/ and then start hadoop. I could find the datanode!

Answer 3

I was having the same problem running a single-node pseudo-distributed instance. Couldn't figure out how to solve it, but a quick workaround is to manually start a DataNode with
hadoop-x.x.x/bin/hadoop datanode

Answer 4

if formatting the tmp directory is not working then try this:

1. first stop all the entities like namenode, datanode etc. (you will be having some script or command to do that)
2. Format tmp directory
3. Go to /var/cache/hadoop-hdfs/hdfs/dfs/ and delete all the contents in the directory manually
4. Now format your namenode again
5. start all the entities then use jps command to confirm that the datanode has been started
6. Now run whichever application you have

Hope this helps.

Answer 5

Run Below Commands in Line:-

1. stop-all.sh (Run Stop All to Stop all the hadoop process)
2. rm -r /usr/local/hadoop/tmp/ (Your Hadoop tmp directory which you configured in hadoop/conf/core-site.xml)
3. sudo mkdir /usr/local/hadoop/tmp (Make the same directory again)
4. hadoop namenode -format (Format your namenode)
5. start-all.sh (Run Start All to start all the hadoop process)
6. JPS (It will show the running processes)

Answer 6

Once I was not able to find data node using jps in hadoop, then I deleted the current folder in the hadoop installed directory (/opt/hadoop-2.7.0/hadoop_data/dfs/data) and restarted hadoop using start-all.sh and jps.

This time I could find the data node and current folder was created again.