Hadoop How to Decommision a DataNode and replace it with another machine?

2
votes

I'm using VMs for my 5 node hadoop cluster (1VM has Name Node, 1VM has JobTracker/SecondaryNameNode/HMaster, and three VMs have DataNodes/TaskTrackers/HRegionServers/Zookeepers), which is of the Cloudera distribution and which I installed manually, as opposed to through Cloudera Manager.

Edit - The diskspace of each VM containing a DataNode is roughly 50-60% full. It would be nice of me to get this done by tomorrow morning but I could get away with 24 hours.

I have to return one of the VMs (specifically one particular DataNode) and replace it with another one (don't ask why). I have the second VM procured and can begin installing whenever I want.

Here is my current strategy:

  1. rsync DataNode's data directory to new Node, as well as the zookeeper's data directory.
  2. rsync all configuration files (core-site.xml,hdfs-site.xml,mapred-site.xml, hbase-site.xml, zoo.cfg)
  3. Ask this question on Stack Overflow

Why number three? The NameNode holds the meta data of the location of all the blocks of all the files stored on HDFS. The HBase Meta table points to the RegionServers which have the HFiles for its data. The Zookeeper server's data on the DataNode is essential, too.

How do I go about instructing the NameNode and HBase/Zookeeper to point to the data on the newly procured VM? What else am I not considering?

Now this is actually a dev environment and I could export the HDFS data and the HBase data using Pig, wipe all the DataNode's and Zookeeper's data directories clean, and import the data back using Pig. Aside from being lame, I believe this would be a good exercise for me.

1
hadoophbasehdfsapache-zookeepersysadmin

1 Answers

0
votes

If you are only replacing a single machine in a reasonably sized cluster, you can generally just turn off the regionserver of the machine you want to decommission wait for the regions to be reassigned and then turn off the datanode and wait for the "Number of Under-Replicated Blocks" to drop to zero in the Hadoop NameNode UI. Then when you add a new machine to the cluster you can run the Hadoop balancer tool to rebalance the data blocks. Then use the HBase shell to run the HBase balancer command. This will rebalance the HBase regions.

If you are decommissioning multiple nodes, you may want to read about draining nodes. You can also mark nodes as decommissioned by using the node exclusions in hdfs-site.xml (dfs.hosts.exclude) and mapred-site.xml (mapreduce.jobtracker.hosts.exclude.filename).

I've found rebalancing regions in HBase to be relatively fast, and not worrisome. Rebalancing HDFS blocks is significantly more time consuming, and so may or may not be worth it depending on the size of your cluster and how full the cluster is.

For what it is worth, I haven't actually tried manually copying the data from one node to another, but I do know that reinserting the node after it has been decommissioned works just fine. (For example, if the machine needed some offline hardware maintenance.) Given that, I would expect your way to work if you replaced the decommissioned machine with a new host with the same DNS and/or IP address. It may also just work the way you've tried, but it seems like a lot more work, and would only be useful if you were trying to minimize the time period when the cluster was under provisioned.