Hadoop 2.2 - Combine Reducer Output

Question

My Map/Reduce job outputs lines of the form

Key1 5
Key2 8
Key3 4
Key1 7
Key3 3

And I would like to aggregate these results to get the sum of the values by key. So something like:

Key1 12 (5 + 7)
Key2 8
Key3 7 (4 + 3)

The naive way would be to simply chain another Map/Reduce job behind the first one but it is inefficient because you have twice more I/O than necessary, the overhead, etc. So I would like to avoid this solution.

The best way to do it would probably be something like map -> reduce1 -> reduce2 where the output of reduce1 is the input of reduce2. Unfortunately, it appears to be impossible to do (see Chaining Multi-Reducers in a Hadoop MapReduce job for example). I have also looked at ChainReducer but it doesn't help either since it doesn't allow several reduce steps.

So how would you go about it? Is there something new in Hadoop 2.X that I could use?

Thank you.

Answer 1

It is not possible to to use multiple reducers without mapper. I would suggest you should try to combine both map-reduces logic in a single map-reduce job, if possible. Otherwise identity mapper would be only solution.

Answer 2

I think you can write a reducer job also.But i haven't tried yet.You can give a try.

And you can also write an identity map only job for this, without specifying

job.setNumReduceTasks(0);

Why because while writing a Map only job with setting job.setNumReduceTasks(0); the output will be in multipe file(output file depends on no of map).

If we are not specifying job.setNumReduceTasks(0); an identity reducer will run by default even though we are not writing a Reducer. And the results get stored in a single part-r-00000 file