How does CopyFromLocal command for Hadoop DFS work?

2
votes

I'm a little confused on how the Hadoop Distributed File System is set up and how my particular setup affects it. I used this guide to set it up http://www.michael-noll.com/tutorials/running-hadoop-on-ubuntu-linux-multi-node-cluster/ using two Virtual Machines on Virtual Box and have run the example (just a simple word count with txt file input). So far, I know that the datanode manages and retrieves the files on its node, while the tasktracker analyzes the data.

1) When you use the command -copyFromLocal, are you are copying files/input to the HDFS? Does Hadoop know how to divide the information between the slaves/master, and how does it do it?

2) In the configuration outlined in the guide linked above, are there technically two slaves (the master acts as both the master and a slave)? Is this common or is the master machine usually only given jobtracker/namenode tasks?

2
hadoophdfs

2 Answers

1
votes

There are lot of questions asked here.

Question 2)

  1. There are two machines
  2. These machines are configured for HDFS and Map-Reduce.
  3. HDFS configuration requires Namenode (master) and Datanodes (Slave)
  4. Map-reduce requires Jobtracker (master) and Tasktracker (Slave)
  5. Only one Namenode and Jobtracker is configured but you can have Datanode and Tasktracker services on both the machines. It is not the machine which acts as master and slave. It is just the services. You can have slave services also installed on machines which contains master services. It is good for simple development setup. In large scale deployment, you dedicate master services to separate machines.

Question 1 Part 2)

  1. It is HDFS job to create file chunk and store on multiple data nodes in replicated manner. You don't have to worry about it.

Question 1 Part 1)

  1. Hadoop file operations are patterned like typical Unix file operations - ls, put etc
  2. Hadoop fs -put localefile /data/somefile --> will copy a localfile to HDFS at path /data/somefile
  3. With put option you can also read from standard input and write to a HDFS file
  4. copyFromLocal is similar to put option except that behavior is restricted to copying from local file system to HDFS
  5. See: http://hadoop.apache.org/common/docs/r0.20.0/hdfs_shell.html#copyFromLocal
0
votes

1)

  • The client connects to the name node to register a new file in HDFS.
  • The name node creates some metadata about the file (either using the default block size, or a configured value for the file)
  • For each block of data to be written, the client queries the name node for a block ID and list of destination datanodes to write the data to. Data is then written to each of the datanodes.

There is some more information in the Javadoc for org.apache.hadoop.hdfs.DFSClient.DFSOutputStream

2) Some production systems will be configured to make the master its own dedicated node (allowing the maximum possible memory allocation, and to avoid CPU contention), but if you have a smaller cluster, then a node which contains a name node and data node is acceptable