Data node capacity increase is not detected by the HDFS

0
votes

I have a virtual hadoop cluster composed of 4 virtual machine (1 master and 3 slaves) and recently added 100GB capacity to my datanodes in cluster. The problem is that HDFS does not detect this increasing. I restart datanode, namenode and also format HDFS but didn't affect. How can I fix this problem?

1
hadoophdfs
By adding you mean what? Did you add the path of your Datanode directory to the hdfs-site.xml? - Abhinav
You don't need to format HDFS only to add a new datanode volume. You do need mount the volume in /etc/fstab and edit the hdfs-site.xml on the datanode with the new mount point - OneCricketeer
@ cricket_007 There is not new Volume! There were a volume with 500GB and now 100GB is added to it. what should I mount in /etc/fstab when the volume does not change? (Don't forget that machines are virtual). - DAVID_ROA
@ Abhinav, "Add" means Increase. Yes, I had added the path of datanode directory in property "dfs.datanode.data.dir". Now, there isn't new directory. I've just changed size of the volume. The problem is HDFS can not detect this increasing and the capacity remains on the same previous value. - DAVID_ROA

1 Answers

1
votes

I'm writing the steps to add datanode/increase hdfs volume size:

  • Create a folder in the root eg. /hdfsdata

  • Create a folder under home eg. /home/hdfsdata

  • Provide permission to 'hdfs' user to this folder: chown hdfs:hadoop -R /home/hdfsdata

  • Provide file/folder permissions to this folder: chmod 777 -R /home/hdfsdata.

  • Mount this new folder: mount --bind /home/hdfsdata/ /hdfsdata/

  • At last, add this newly created directory to the hdfs-site.xml under the property "dfs.datanode.data.dir" with a comma separated value.

When i cd to /hadoop/hdfs/data

After the above steps, restart the HDFS service and you have your capacity increased.