Hadoop - is there a locality awareness check for Reduce task node assignment

Question

All the books and blogs I have read so far does not provide much info about reduce task assignment. It looks like the reduce task assignments to available slots are random.

This does not make sense as shuffling data across network without considering data(map) locality goes against hadoop design principles.

There is a good chance (not a definite possibility) blocks from the same file are placed within the same rack or nearby racks. So, map tasks for those splits/blocks will also be in those racks (most of the times).

If this is a possible scenario, why not try to assign reduce tasks to slots in the same rack/s as map tasks?

Wouldn't this improve performance in a 1000+ node cluster? Particularly when the input is a sequence or map file.

Can anyone please confirm that reducers are placed randomly is correct (the definitive edition book says so)? If yes, why that decision was made? If i am wrong? then the logic of how reducers are assigned...links to some docs explaining that logic would be nice too.

Thanks a lot in advance.

Arun

Answer 1

There is no way to give a reduce task "data locality". Each reducer gets data from every mapper (not from a file or blocks like you are saying). This is because every mapper could potentially write a key that will go to any number of the reducers (for more information look up the Partitioner). So, with 1000 nodes, the best you could do is make 1/1000 of the data local on average, so randomly distributing reduce tasks is the best you can do.

The concern of how much data is going between the mappers and reducers in the shuffle step is most definitely an important consideration. That's why you want to filter as much as possible up front and use proper encoding for data types to minimize the amount of data going through.

Answer 2

I think having a combiner step, between map and reduce, is how you try to maximize locality as an integral part of reduce (at least conceptually) because it does some "local" processing before passing data onto the final reducers. It is impossible for reduce to stay local, as Donald mentioned.