Know the disk space of data nodes in hadoop?

问题

Is there a way or any command using which I can come to know the disk space of each datanode or the total cluster disk space?

I tried the command

dfs -du -h /

but it seems that I do not have permission to execute it for many directories and hence cannot get the actual disk space.

回答1:

From UI:

http://namenode:50070/dfshealth.html#tab-datanode

---> which will give you all the details about datanode.

From command line:

To get disk space of each datanode:

sudo -u hdfs hdfs dfsadmin -report

---> which will give you the details of entire HDFS and the individual datanodes OR

sudo -u hdfs hdfs dfs -du -h /

---> which will give you the total disk usage of each folder under root / directory

回答2:

You view the information about all datanodes and their disk usage in the namenode UI's Datanodes tab. Total cluster disk space can be seen in the summary part of the main page.

http://namenode-ip:50070

回答3:

If you are using Hadoop cluster configured as simple security, you can execute the below command to get the usage of data nodes.

export HADOOP_USER_NAME=hdfs ; 

* Above command can be used to get admin privilege in simple security, If you are using any other user for hdfs admin, replace hdfs with the respective hdfs admin user.

hadoop dfsadmin -report 

Alternate option is to login to respective datanode and execute the below unix command to get disk utilization of that server.

df -h

回答4:

Hadoop 3.2.0:

hduser@hadoop-node1:~$ hdfs dfs -df
Filesystem                          Size          Used     Available  Use%
hdfs://hadoop-node1:54310  3000457228288  461352007680  821808787456   15%
hduser@hadoop-node1:~$

For human-readable numbers, use:

hduser@hadoop-node1:~$ hdfs dfs -df -h
Filesystem                  Size     Used  Available  Use%
hdfs://hadoop-node1:54310  2.7 T  429.7 G    765.4 G   15%
hduser@hadoop-node1:~$ 

来源：https://stackoverflow.com/questions/45808525/know-the-disk-space-of-data-nodes-in-hadoop