Where Does the HDFS Account for Triple Replication in Usage Reports?

Question

In the latest version of most Hadoop distributions, the HDFS usage reports seem to report on space without accounting for the replication factor, correct?

When one looks at the Namenode Web UI and/or runs the 'hadoop dfsadmin -report' command, one can see a report that looks something like this:

Configured Capacity: 247699161084 (230.69 GB)
Present Capacity: 233972113408 (217.9 GB)
DFS Remaining: 162082414592 (150.95 GB)
DFS Used: 71889698816 (66.95 GB)
DFS Used%: 30.73%
Under replicated blocks: 40
Blocks with corrupt replicas: 6
Missing blocks: 0

Based on the machine sizes of this cluster, it seems that this report does NOT account for triple replication... I.E. If I place a file on the HDFS, I should account for the triple replication myself.

For example, if I placed a 50GB file on the HDFS, would my HDFS be dangerously close to full (since it seems that file would be replicated 3 times, using up the 150GB that currently remain)?

how do you infer from this report that HDFS does not account for replication? — Chaos
you can try loading a 1 GB file and run dfsadmin report to see if the DFS Used increases by 3 GB — Chaos
Ahh good old empirical experimentation... I will attempt to do this and post the results, but there are many users on my Hadoop cluster (and I'd prefer to avoid configuring another cluster if someone has one readily available)-- so they might not be accurate. I was however hoping for some kind of "debug" report or additional detail feature that shows how much space is actually available and remaining assuming everything is properly triple replicated — depthfirstdesigner
As far as I know, dfsadmin report does consider replication factor when calculating dfs used — Chaos
So I currently have 4 nodes that contribute approximately 58GB each to the HDFS, which gives 232GB in total (Configured capacity in dfsadmin report above), note that this value does not consider the replication factor... This leads me to believe that the DFS Used is a raw number that does have the triple replication 'cost' built-in to it (because it simply counts the bytes that are used), but that DFS remaining will be a raw number that does not have triple replication built-in. It's might seem a pedantic point, but it's one I'd like to have answered with certainty. — depthfirstdesigner

Chaos Chaos · Accepted Answer · 2013-07-25T20:10:57

Let us define clearly what each of these terms mean.

Configured Capacity: It is the total capacity available to HDFS for Storage. So if you have 4 nodes and each node has 50 GB capacity, the configured capacity will be 200 GB. Replication factor is irrelevant in case of configured capacity.
DFS Used: This is the amount of storage space that has been used up by HDFS. Divide DFS Used by your replication factor to get the actual size of your files stored without replication. So if your DFS used is 60 GB, and your replication factor is 3, the actual size of your files is 60/3 = 20 GB.
DFS Remaining: This is the amount of storage space still available to the HDFS. If you have 150 GB remaining storage space, that mean you can store upto 150/3 = 50 GB of files without exceeding your Configured Capacity (assuming replication factor = 3).
Present Capacity: The amount of storage space available for storing user files after allocating space for metadata. The difference:(Configured capacity - Present capacity) is used for storing file system metadata. and inode information.

Hope this clears it up.

Where Does the HDFS Account for Triple Replication in Usage Reports?

2 Answers