How to copy files from HDFS to remote HDFS

0
votes

I want to copy files from my Hadoop cluster to the remote cluster.

I have the hadoop_conf file in the remote cluster and can access it by setting HADOOP_CONF_DIR.

I know the IP and port of the remote name node.

I want to copy the file through the namespace as ex below.

ex) hadoop fs -cp hdfs://MyNamespace/path/file hdfs://RemoteNamespace/path/file

However, if I do not configure hadoop_conf_dir, I do not know the remote namespace, and if I set the information of remote cluster in hadoop_conf_dir, I can not access my cluster's namespace.

Please let me know how to do it.

1
hadoophdfs

1 Answers

1
votes

The typical way to copy between clusters is using distcp.

$ hadoop distcp hdfs://nn1:8020/foo/bar hdfs://nn2:8020/bar/foo

See DistCp Version2 Guide for more information.