Sorting Mapper output by key and then by value

3
votes

I am trying to write a sample Map Reduce program whose Mapper output looks like:

1/1/2012        15:11:46
1/1/2012        19:09:26
1/1/2012        14:01:25
1/1/2012        17:32:26
1/1/2012        17:41:00
1/1/2012        19:35:38
1/1/2012        14:28:10
1/1/2012        15:45:55

I want my input to the reducer sorted by key and then by value. By default, Hadoop framework sorts the mapper output only by key.

I think I should be using the Secondary Sort to accomplish this task but not sure how to use this.

Can anyone please help me with this?

2
hadoopmapreduce
You need to use secondary sorting. For an example have a look at: vangjee.wordpress.com/2012/03/20/… - Lorand Bendig
You are correct, you need to use secondary sort for this. You can find mapreduce code for secondary sort online . - Chaos

2 Answers

1
votes

At a high level:

  • Make your key a concatenation of the current key and value. Keep the value the same.
  • Create a Grouping Comparator that takes two keys (which are concatenations), extracts just the dates and returns a comparison of the two dates. This makes all records with the same date get passed on a single call to reduce().
  • Specify your grouping comparator in your job driver with all of the other job and configuration settings.

Note that your date values as shown won't be sorted by date lexically - you want the year to be first.

EDIT: It occurs to me that you'll also probably have to write a partitioner since you want make sure that keys that apparently have different values (but which are all on the same day) get sent to the same partition.

0
votes

Have a custom Hadoop WritableComparable, as in TextPair pair example TEXT PAIR ,

  • use it as KEY with Have your Date as first element in TextPair class, and time as second.

Incase you DONT want to allocate Diffrent Reducer, for same Date with different TIME , USE a custom partitioner, which would partition based on Date Alone