MapReduce: Passing external jar files using libjars option does not work

Question

My map reduce program needs external jar files. I am using the "-libjars" option to provide those external jar files -

I used Tool, Configured and ToolRunner Utilities provided by hadoop.

  public static void main(String[] args)throws Exception {
         int res = ToolRunner.run(newConfiguration(), new MapReduce(),args);
         System.exit(res);
  }

  @Override
  public int run(String[] args) throwsException {
         // Configuration processed by ToolRunner
   Configuration conf = getConf();
   Job job = new Job (conf, "MapReduce");
   ....
   }

When I tried to run the job -

$ Hadoop jar myjob.jar jobClassName -libjars external.jar

It threw the following exception.

12/11/21 16:26:02 INFO mapred.JobClient: Task Id : attempt_201211211620_0001_m_000000_1, Status : FAILED Error: java.lang.ClassNotFoundException: org.joda.time.format.DateTimeFormatterBuilder

I have been trying to resolve it for a while. Nothing seems to work so far. I am using CDH 4.1.1.

I found this question while having the same problem. If I figure it out I'll post, please do the same if you find the exact resolution first. — David Parks
Check the answer posted here: stackoverflow.com/questions/6890087/… — Vladimir Kroz

Diego Pino Diego Pino · Accepted Answer · 2012-11-23T23:38:04

It seems cannot find JodaTime. Open /etc/hbase/hbase-env.sh and add your extra jar to HADOOP_CLASSPATH.

export HADOOP_CLASSPATH="<extra_entries>:$HADOOP_CLASSPATH"

Another, less efficient and sometimes not possible, idea is to copy your requited jar to /usr/share/hadoop/lib.

MapReduce: Passing external jar files using libjars option does not work

2 Answers