Hadoop library conflict at mapreduce time

1
votes

I have a jar that uses the Hadoop API to launch various remote mapreduce jobs (ie, im not using the command-line to initiate the job). The service jar that executes the various jobs is built with maven's "jar-with-dependencies".

My jobs all run fine except one that uses commons-codec 1.7, I get:

FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.encodeAsString([B)Ljava/lang/String;

I think this is because my jar is including commons-codec 1.7 whereas my Hadoop install's lib has commons-codec 1.4 ...

Is their any way to instruct Hadoop to use the distributed commons-codec 1.7 (I assume this is distributed as a job dependency) rather than the commons-codec 1.4 in the hadoop 1.0.3 core lib?

Many thanks!

Note: Removing commons-codec-1.4.jar from my Hadoop library folder does solve the problem, but doesn't seem too sane. Hopefully there is a better alternative.

1
javamavenhadoop

1 Answers

0
votes

Two approaches:

  • You should be able to exclude commons-codec from within the hadoop dependency and add another explicit dependency for commons-codec
  • Try setting the scope to provided so that none of the hadoop jars get included. This assumes that those jars would be in the runtime class path.