Run Hadoop Application

3
votes

  1. Can a Hadoop App be booted on any one of the cluster nodes, irrelvantly to the node type?

  2. Should the App be booted only on one of the nodes?

  3. Does Hadoop move the code to the proper nodes near to the data?

  4. Is Java RMI used for the purpose of moving the code?

@EDIT

On each Hadoop installed node, who are the master nodes should be explicitly assigned, by setting fs.default.name of core-site.xml for the Namenode and mapred.job.tracker of mapred-stie.xml for the Jobtracker.

However, the slave nodes Datanode and Tasktracer can't be explicitly assigned, can they?

Instead, they dynamically join the cluster when its Hadoop dameon starts, or leave the cluster when the daemon stops , right?

If a Hadoop installed node is neither assigned to a master nor its hadoop daemon runs, it can still launch a Hadoop App, right?

@2nd EDIT

It seems that the slave nodes should also be explicitly listed in conf/slaves file on the master nodes in order to collectively be started or stopped over there.

1
hadoop
#1 is actually a common question for new developers in my larger clusters. There is a common misconception that you want to run jobs on the jobtracker node, which is one of the worst places to run a job. - Donald Miner

1 Answers

4
votes

  1. Hadoop jobs can fire from anywhere, as long as the core-site.xml configuration is set properly. This doesn't have to be a data node or a master node. On larger clusters it us customary to have a "query node" that doesn't have any of the Hadoop daemons running on it.

  2. It's not a big deal to have the job fire on specific node, unless your program does some sort of heavy pre- or post-computation outside of MapReduce. If they are, you want to keep these on their own node. Running the jobs on a data node is better than running the job on the master nodes.

  3. Hadoop replicates the jar files all over the place so the code is available locally. I don't think this uses RMI, but I could be wrong. In the end, it's all abstracted away from the user and is not something you need to pay attention to.