Hadoop: ClassNotFoundException - org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat

1
votes

I'm facing ClassNotFoundException, when I run my job for the class org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat.

I tried to pass the additional jar files with -libjars, still I am facing the same issue. Any suggestions will be greatly helpful. Thanks in advance.

Below is the command I am using and exception I am facing!

hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hcatalog/rcfile/RCFileMapReduceOutputFormat
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.run(OmnitureToRCFileJob.java:91)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at com.cloudera.sa.omniture.mr.OmnitureToRCFileJob.main(OmnitureToRCFileJob.java:131)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
Caused by: java.lang.ClassNotFoundException: org.apache.hcatalog.rcfile.RCFileMapReduceOutputFormat
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 8 more

I implemented ToolRunner as well, below is the code which confirms that!

public class OmnitureToRCFileJob extends Configured implements Tool {

    public static void main(String[] args) throws Exception {

        OmnitureToRCFileJob processor = new OmnitureToRCFileJob();
        String[] otherArgs = new GenericOptionsParser(processor.getConf(), args).getRemainingArgs();

        System.exit(ToolRunner.run(processor.getConf(), processor, otherArgs));
    }

}
3
hadoopmapreduce
@Martin Prikryl Hi Martin, how are you able to edit the question.. ? is it like anyone can edit the questions we posted on StackOverFlow.. ? i am new t SOF.. can you tell me ? – Raja
There's "edit" link below tags (above comments). I believe anyone can edit his own posts. – Martin Prikryl
yeah, everyone can edit his own posts. but how are you able to edit my post... ? that's confusing me!! – Raja
oh! i dont know that... thank you Martin! – Raja

3 Answers

0
votes

Did you try running by giving full path of "hcatalog-core-0.5.0-cdh4.4.0.jar" jar file in your below line.

hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir

or

Below configuration should also work for you

$ export LIBJARS= <fullpath>/hcatalog-core-0.5.0-cdh4.4.0.jar
$hadoop jar MyJob.jar MyDriver -libjars ${LIBJARS} inputDir OutputDir
0
votes

If you look at hadoop command documentation, you can see that -libjars is a generic option. For parsing generic option, you got to override the ToolRunner.run() method in your driver class as follows :

public class TestDriver extends Configured implements Tool {
    @Override
    public int run(String[] args) throws Exception {
        Configuration conf = getConf();

        # Job configuration details            
        # Job submission  

        return 0;
    }        
}

public static void main(String[] args) throws Exception {
    int exitCode = ToolRunner.run(new TestDriver(), args);
    System.exit(exitCode);
}

I'nk you are getting this exception from your driver code itself. Setting hcatalog-cor*.jar using -libjars option may not be available in client JVM(JVM in which driver code runs). Better you need to set this jar in HADOOP_CLASSPATH environment variable before executing the same using hadoop jar as follows

export HADOOP_CLASSPATH=${HADOOP_CLASSPATH}:<PATH-TO-HCAT-LIB>/hcatalog-core-0.5.0-cdh4.4.0.jar;
hadoop jar MyJob.jar MyDriver -libjars hcatalog-core-0.5.0-cdh4.4.0.jar inputDir OutputDir
0
votes

Id had the same problem but found out the jar command doesn't accept the --libjars argument. "Specify comma separated jar files to include in the classpath. Applies only to job." --> Hadoop Cli Generic Options

Instead you should use the env vars to add additional or replace jars.

export HADOOP_USER_CLASSPATH_FIRST=true export HADOOP_CLASSPATH="./lib/*"