How to reset “Reserved Space for Replicas” without restart Hadoop services

Question

How to find why "Reserved Space for Replicas" constantly increasing and how to limit space for this type of cache?

We found that the "Reserved Space for Replicas" relate to "Non DFS" space and it can increase constantly up to rebooting Data Node-s. We didn't find how to limit space allocated for the "Reserved Space for Replicas" :(

We thought that dfs.datanode.du.reserved can control "Reserved Space for Replicas" but it is not.

So our question is How to control a space size allocated for the "Reserved Space for Replicas" in Hadoop ?

Answer 1

For people who faced with that type of problem. First of all you should understand the nature of the problem. To do that please read the description of the following issues:

• https://issues.apache.org/jira/browse/HDFS-8072
• https://issues.apache.org/jira/browse/HDFS-9530
• https://issues.apache.org/jira/browse/HDFS-6489

The following links will be useful to understand what block replica is:

• https://blog.cloudera.com/understanding-hdfs-recovery-processes-part-1/
• https://blog.cloudera.com/understanding-hdfs-recovery-processes-part-2/

Solutions

1. Find the wrong software which often broke connection with Hadoop during the write or append processes
2. Try to change replication policy (risky)
3. Update Hadoop up to late version

You can't reset “Reserved Space for Replicas” without restart Hadoop services!