sqoop integration with hadoop throw ClassNotFoundException

0
votes

I am new in word of hadoop and sqoop. I installed hadoop 2.7.3 (pseudo mode) and its working fine on my system.

I want integration with sqoop. I am using sqoop sqoop-1.99.7-bin-hadoop200.

1) I extract tar file and move extracted content into /usr/local/sqoop

2) Set Sqoop path into .bashrc file.

Snapshot of **.bashrc** file 3) go to /usr/local/sqoop/server/lib/sqoop.sh server start

and get following error message.

hadoop_usr@sawai-Lenovo-G580:/usr/local/sqoop/server/lib$ sqoop.sh server start
Setting conf dir: /usr/local/sqoop/conf
Sqoop home directory: /usr/local/sqoop
Starting the Sqoop2 server...
0    [main] INFO  org.apache.sqoop.core.SqoopServer  - Initializing Sqoop server.
34   [main] INFO  org.apache.sqoop.core.PropertiesConfigurationProvider  - Starting config file poller thread
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/configuration/Configuration
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<init>(DefaultMetricsSystem.java:38)
    at org.apache.hadoop.metrics2.lib.DefaultMetricsSystem.<clinit>(DefaultMetricsSystem.java:36)
    at org.apache.hadoop.security.UserGroupInformation$UgiMetrics.create(UserGroupInformation.java:121)
    at org.apache.hadoop.security.UserGroupInformation.<clinit>(UserGroupInformation.java:237)
    at org.apache.sqoop.security.authentication.SimpleAuthenticationHandler.secureLogin(SimpleAuthenticationHandler.java:39)
    at org.apache.sqoop.security.AuthenticationManager.initialize(AuthenticationManager.java:98)
    at org.apache.sqoop.core.SqoopServer.initialize(SqoopServer.java:57)
    at org.apache.sqoop.server.SqoopJettyServer.<init>(SqoopJettyServer.java:67)
    at org.apache.sqoop.server.SqoopJettyServer.main(SqoopJettyServer.java:177)
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.Configuration
    at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    ... 9 more

4) So when i am looking for jars in lib then i found. following jar list.

jar in **lib** directory

Please can you tell me which jar is missing now. So am able to run sqoop and start my learning.

Thanks.

4
javahadoopsqoopsqoop2
few google guide suggest me to put hadoop path into sqoop-env file but in sqoop/conf there is no file similar to sqoop-env. – sawai singh
This problem resolved i download a jar file from here commons.apache.org/configuration and /usr/local/sqoop/server/lib – sawai singh

4 Answers

3
votes

I got the same issue

to solve this,I have set below variables in seperate terminal and it works

export HADOOP_MAPRED_HOME=/opt/hadoop/hadoop-3.1.2/share/hadoop/mapreduce
export HADOOP_COMMON_HOME=/opt/hadoop/hadoop-3.1.2/share/hadoop/common
export HADOOP_HDFS_HOME=/opt/hadoop/hadoop-3.1.2/share/hadoop/hdfs
export YARN_HOME=/opt/hadoop/hadoop-3.1.2/share/hadoop/yarn
2
votes

Copy the jar files from below to $SQOOP_HOME/server/lib :

cp -R $HADOOP_HOME/share/hadoop/common/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/common/lib/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/hdfs/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/hdfs/lib/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/mapreduce/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/mapreduce/lib/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/yarn/* $SQOOP_HOME/server/lib/
cp -R $HADOOP_HOME/share/hadoop/yarn/lib/* $SQOOP_HOME/server/lib/

And remove duplicate jars with different versions.

Did face the same issue. I have configured the same versions as yours. These steps are working for me, right now.

Good Luck ! We are on the same page !

1
votes

This got solved by setting the following two environment variables in my .bashrc file.

export HADOOP_COMMON_HOME=/usr/local/hadoop/share/hadoop/common

export HADOOP_MAPRED_HOME=/usr/local/hadoop/share/hadoop/mapreduce

basically ensuring that the common libraries mentioned in the above answers are available through these environment variables for sqoop.sh

-1
votes

Please make sure if hadoop-common-0.xx.jar in classpath to resolve classnotfound Exception for configuration class.

hadoop-common-0.xx.jar will be available in /lib.