Find out actual disk usage in HDFS

7
votes

Is there a way to find out how much space is consumed in HDFS? I used 

hdfs dfs -df

but it seems to be not relevant cause after deleting huge amount of data with

hdfs dfs -rm -r -skipTrash

the previous comand displays changes not at once but after several minutes (I need up-to-date disk usage info).

3
hadoophdfsbigdatadiskspace
No solution?? Plase comment the answer that show a valid solution. - Peter Krauss

3 Answers

7
votes

To see the space consumed by a particular folder try:

hadoop fs -du -s /folder/path

And if you want to see the usage, space consumed, space available, etc. of the whole HDFS:

hadoop dfsadmin -report
4
votes

hadoop cli is deprecated. Use hdfs instead.

Folder wise :

sudo -u hdfs hdfs dfs -du -h /

Cluster wise :

sudo -u hdfs hdfs dfsadmin -report
-1
votes
hadoop fs -count -q /path/to/directory