Hadoop data locality, counter-intuitive observation

Question

Can anyone help me understand following observation that is opposite to my understand of Hadoop data locality.

A Hadoop cluster with 3 nodes:

master: 10.28.75.146

slave1: 10.157.6.202

slave2: 10.31.130.224

run a task successfully. From job console:

Task Attempts:attempt_201304030122_0003_m_000000_0
Machine: /default-rack/10.31.130.224<p>
Task log: INFO: consuming hdfs://10.28.75.146:9000/input/22.seq

We know 224 node is processing /input/22.seq data. By command:

$hadoop fsck /input -files -blocks  -locations |grep -A 1 "22.seq"
/input/22.seq 61731242 bytes, 1 block(s):  OK
0. blk_-8703092405392537739_1175 len=61731242 repl=1 [10.157.6.202:9200]

22.seq fits in one block which is smaller than default HDFS block size (64MB) and not replicated to other node.

Question: since 22.seq is not local to 224 node, why Hadoop assigns 224 node processing data remotely on 202?

Note: this is not an exception. I notice many data files are fetched remotely, and observe huge network traffic on eth0 at both nodes. I am expecting near-zero traffic between two nodes, since all my data files are <64MB, and data should processed locally.

FYI: This is observed on Amazon's AWS EMR.

Engineiro Engineiro · Accepted Answer · 2013-04-03T16:12:15

I am not sure if this will answer your question fully, but I will attempt to shine some light.

The network traffic you encountered above may have been influenced by the process by which the mapreduce framework submits a job; part of which transfers by default 10 copies of your job jar and all libraries contained therein across the cluster (in cases like yours where there are not 10 nodes I am not sure how it would behave): there are heatbeats and getting input split info and reporting progress which seem like small bandwidth operation although I am ignorant about the specifics on their network resource consumption.

Regarding the job you are running: If it is a map only job then Hadoop tries (tries because there may be resource limiting factors running on the data-local node) for data locality optimization and runs the job where the input split is located. It sounds like in your case, the file is less than the default 64MB so 1 split should equal your data which in turn should result in one map since the number is maps is directly proportional to the number of splits you have, but if your job is a Map and Reduce job then the network traffic may be picking up some of the reduce copy and sort phase HTTP network traffic which can end up on separate nodes.

N Input Splits = N Maps --output--> M partitions = M Reducers

Of course the network traffic and data locality optimizations are dependent on the availability of the nodes resources so your test assumptions should take this into consideration.

Hope I was a tiny bit helpful.

Hadoop data locality, counter-intuitive observation

2 Answers