Behavior of HDFS High Availability

Question

How does distributed copy (distcp) work between two clusters when NameNode (NN) fails in High Availability (HA) configuration.

Will that job fail due to different IP address of name node and the standby node?

Answer 1

Depending on the configuration of your HDFS HA and if Automatic Failover is implemented, it might work (I personally haven't tested the specific command during a failover).

Another important part is that you are using names for the services and DNS is properly setup and configured for all involved nodes (you should never use direct IP addresses).

Answer 2

Yashwanth, In an HA Hadoop cluster, it is not recommended to use active name node in the distcp commands. A simple answer to your question is Yes, if you hardcode Namenode IP or DNS in the distcp command. In an HA hadoop cluster you need to use cluster name in of IP in the distcp command.