Is it better to use the mapred or the mapreduce package to create a Hadoop Job?

46
votes

To create MapReduce jobs you can either use the old org.apache.hadoop.mapred package or the newer org.apache.hadoop.mapreduce package for Mappers and Reducers, Jobs ... The first one had been marked as deprecated but this got reverted meanwhile. Now I wonder whether it is better to use the old mapred package or the new mapreduce package to create a job and why. Or is it just dependent on whether you need stuff like the MultipleTextOutputFormat which is only available in the old mapred package?

3
hadoopmapreduce
but this got reverted meanwhile are you sure? – Thomas Jungblut
E.g. Interface Mapper in package org.apache.hadoop.mapred.lib in r0.21.0 is not marked as deprecated while it is marked as deprecated in r0.20.2. – momo13

3 Answers

45
votes

Functionality wise there is not much difference between the old (o.a.h.mapred) and the new (o.a.h.mapreduce) API. The only significant difference is that records are pushed to the mapper/reducer in the old API. While the new API supports both pull/push mechanism. You can get more information about the pull mechanism here.

Also, the old API has been un-deprecated since 0.21. You can find more information about the new API here.

As you mentioned some of the classes (like MultipleTextOutputFormat) have not been migrated to the new API, due to this and the above mentioned reason it's better to stick to the old API (although a translation is usually quite simple).

14
votes

Both the old and new APIs are good. The new API is cleaner though. Use the new API wherever you can, and use the old one wherever you need specific classes that are not present in the new API (like MultipleTextOutputFormat)

But do take care not to use a mix of the old and new APIs in the same Mapreduce job. That leads to weird problems.

2
votes

Old API (mapred)

  1. Exists in Package org.apache.hadoop.mapred

  2. Provide A map/reduce job configuration.

  3. Reduces values for a given key, based on the Iterator
  4. Package Summary

New API (mapreduce)

  1. Exists in Package org.apache.hadoop.mapreduce

  2. Job configuration is done by separate class, Called JobConf which is extension of Configuration
    Class

  3. Reduces values for a given key, based on the Iterable

  4. Package Summary