Hadoop: Compressing output of Map-only job

1
votes

I have a a map-only job that outputs in TextOutputFormat.

I currently see three ways of compressing my output:

  1. by defining map to compress through mapred.compress.map.output.*

  2. by defining output to compress through mapred.output.compression.*

  3. by defining the TextOutputFormat to compress through TextOutputFormat.setCompressOutput()

  4. by composing one or more of the first 3 possibilities.

Any insight on how to do this properly?

1
javacompressionhadoopmapreducehdfs

1 Answers

0
votes

Option 1 is ONLY for intermediate output, Option 2 works as well but is more of a mapred-site.xml approach, Option 3 is the way to go.