Some datanodes still showing block pool used after clearing the HDFS

Question

Hadoop Version: 2.7.3 Datanodes: 32 BlockSize: 512m Replication: 3

I Cleared all the data in HDFS using the command hdfs dfs -rm -r /*

After clearing the HDFS cluster, few datanodes still showing the Block pool used, though blocks are zero.

The screen shot of hdfs webui

The output of hdfs fsck / is as follows

Connecting to namenode via http://ip-10-0-2-7:50070/fsck?ugi=ubuntu&path=%2F
FSCK started by ubuntu (auth:SIMPLE) from /10.0.2.7 for path / at Tue Jul 18 04:34:19 UTC 2017
Status: HEALTHY
 Total size:    0 B
 Total dirs:    1
 Total files:   0
 Total symlinks:        0
 Total blocks (validated):  0
 Minimally replicated blocks:   0
 Over-replicated blocks:    0
 Under-replicated blocks:   0
 Mis-replicated blocks:     0
 Default replication factor:    3
 Average block replication: 0.0
 Corrupt blocks:        0
 Missing replicas:      0
 Number of data-nodes:      32
 Number of racks:       1
FSCK ended at Tue Jul 18 04:34:19 UTC 2017 in 1 milliseconds
The filesystem under path '/' is HEALTHY

I know the block deletion is asynchronous process, but the datanodes which are having the Block Pool Used not set to 0, taking a lot of time to release the blocks. Can anybody help me figure out, why on those servers, block deletion is slow, or any other issue because of which the blocks are being used.

have you tried the skiptrash option while deleting files? eg. hdfs dfs -rm [-skipTrash] URI — vasanth

Chris Nauroth Chris Nauroth · Accepted Answer · 2017-07-18T05:19:24

There are several potential causes for this:

The cluster software was upgraded recently using the HDFS Rolling Upgrade feature, and the upgrade has not been finalized. During a rolling upgrade, file deletions do not actually delete the underlying block files at the DataNode. Instead, the block files are maintained, so that if the operator chooses to roll back the upgrade, the cluster's data can be reverted back to pre-upgrade state. As a consequence, during the rolling upgrade window, DataNode metrics such as "Block pool used" continue to show space consumed. An easy way to check if a rolling upgrade is in progress is to go to the NameNode web UI. The Overview tab will show a message about "Rolling upgrade started". To finalize the upgrade, run hdfs dfsadmin -rollingUpgrade. Afterwards, DataNodes will start deleting block files (asynchronously).
A user of the cluster made use of HDFS Snapshots. Snapshots maintain the state of a file system path at the moment the snapshot was created. To support this, DataNodes may need to keep blocks for previously deleted files, in case a user chooses to read an old snapshot containing the data. Running hdfs fsck with the -includeSnapshots argument can show if there are snapshots in use.
Due to some kind of I/O error (failed disk, permissions violation on underlying block files), the DataNode is failing to delete the block files. This is unlikely, but if there is a problem, then the DataNode logs would show more information.

Some datanodes still showing block pool used after clearing the HDFS

1 Answers