Oozie error - while importing data from mysql to hdfs

1
votes

I am trying to import data to hdfs from mysql through sqoop. I am able to run sqoop through shell and it is working fine . but when I am invoking through oozie. it is arising following error and getting fail. I have also included jars.

Now I am trying this at my home - CDH 5.3 single node in virtual box but error is still the same

I know this has been discussed earlier but still no help..

Main error is:

ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Job properties file is as below, present in local environment

nameNode=hdfs://localhost:8020
jobTracker=localhost:8032

oozie.use.system.libpath=true
oozie.wf.application.path=${nameNode}/user/harry/

Workflow.xml file is as below, which is present in HDFS under ${nameNode}/user/harry/

<workflow-app name="Trend_Analysi_Scoop_Script" xmlns="uri:oozie:workflow:0.4">
<start to="sqoop-c68d"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="sqoop-c68d">
    <sqoop xmlns="uri:oozie:sqoop-action:0.2">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <arg>import</arg>
        <arg> --connect</arg>
        <arg>jdbc:mysql://localhost/oozie_training</arg>
        <arg>--username</arg>
        <arg>root</arg>
        <arg>--password</arg>
        <arg>cloudera</arg>
        <arg>--table</arg>
        <arg>user</arg>
        <arg>--target-dir</arg>
        <arg>/training/sqoop</arg>
        <arg>-m</arg>
        <arg>1</arg>
    </sqoop>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

I have included all the necessary jars under below path 

[cloudera@quickstart ~]$ hadoop fs -ls /user/oozie/share/lib/lib_20141218070949/sqoop
Found 75 items
-rw-r--r--   1 root  oozie     960374 2016-05-26 22:00 /user/oozie/share/lib/lib_20141218070949/sqoop/mysql-connector-java-5.1.34-bin.jar
-rw-r--r--   1 oozie oozie      16924 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/oozie-sharelib-sqoop-4.0.0-cdh5.3.0.jar
-rw-r--r--   1 oozie oozie    1010790 2014-12-18 07:09 /user/oozie/share/lib/lib_20141218070949/sqoop/sqoop-1.4.5-cdh5.3.0.jar

I have also restarted the oozie server after adding jars.

Error logs i am getting are as below.

2016-05-24 10:37:17,994  INFO ActionStartXCommand:520 - SERVER[***]             USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] Start action [0000001-160524103133053-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action status=DONE
2016-05-24 10:37:17,997  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] [***0000001-160524103133053-oozie-oozi-W@:start:***]Action updated in DB!
2016-05-24 10:37:18,164  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@:start:
2016-05-24 10:37:18,165  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:18,188  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Start action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:26,080  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:26,106  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] checking action, hadoop job ID [job_1462295072181_0163] status [RUNNING]
2016-05-24 10:37:26,108  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action status=RUNNING
2016-05-24 10:37:26,114  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] [***0000001-160524103133053-oozie-oozi-W@sqoop-c68d***]Action updated in DB!
2016-05-24 10:37:26,184  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@sqoop-c68d
2016-05-24 10:37:37,181  INFO CallbackServlet:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] callback for action [0000001-160524103133053-oozie-oozi-W@sqoop-c68d]
2016-05-24 10:37:37,314  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Trying to get job [job_1462295072181_0163], attempt [1]
2016-05-24 10:37:37,388  INFO SqoopActionExecutor:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] action completed, external ID [job_1462295072181_0163]
2016-05-24 10:37:37,395  WARN SqoopActionExecutor:523 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] Launcher ERROR, reason: Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]
2016-05-24 10:37:37,484  INFO ActionEndXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] ERROR is considered as FAILED for SLA
2016-05-24 10:37:37,679  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] Start action [0000001-160524103133053-oozie-oozi-W@Kill] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action status=DONE
2016-05-24 10:37:37,683  INFO ActionStartXCommand:520 - SERVER[***] USER[oozie] GROUP[-] TOKEN[] APP[Trend_Analysi_Scoop_Script] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] [***0000001-160524103133053-oozie-oozi-W@Kill***]Action updated in DB!
2016-05-24 10:37:37,930  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@Kill] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W@Kill
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi-W
2016-05-24 10:37:37,931  INFO WorkflowNotificationXCommand:520 - SERVER[***] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-160524103133053-oozie-oozi-W] ACTION[0000001-160524103133053-oozie-oozi-W@sqoop-c68d] No Notification URL is defined. Therefore nothing to notify for job 0000001-160524103133053-oozie-oozi- W@sqoop-c68d

I think oozie is not picking up the share/lib, if so it would be really great if someone could help me out on the same

Any help would be appreciated.

2
sqoopoozie
Can you check the logs from launcher job? You should find more information there regarding the failure? – YoungHobbit
Where i can get a launcher job logs ? – Hardeep Ajmani

2 Answers

0
votes

Looks like there is some issue with your "command" tag in the sqoop action. Since it is already a sqoop action "/usr/hdp/2.4.0.0-169/sqoop/bin/sqoop" is not needed. Refer the examples in the below link to correct them. And try rerunning the job again.

https://oozie.apache.org/docs/4.0.0/DG_SqoopActionExtension.html#Sqoop_Action

0
votes

You are using the --query option in the command element, which is not supported in the oozie. You need to use the arg tag and provide all the arguments individually. The command tag is split by whitespace so it will not work with --query option. Here is the sample example:

        <arg>import</arg>
        <arg>--connect</arg>
        <arg>jdbc:mysql://host:3306/oozie</arg>
        <arg>--username</arg>
        <arg>user</arg>
        <arg>--password</arg>
        <arg>pwd</arg>
        <arg>--driver</arg>
        <arg>com.mysql.jdbc.Driver</arg>
        <arg>--query</arg>
        <arg>"select BUNDLE_ACTIONS.* from BUNDLE_ACTIONS where $CONDITIONS"</arg>
        <arg>--hcatalog-table</arg>
        <arg>bundle_actions</arg>
        <arg>--hcatalog-database</arg>
        <arg>oozie</arg>
        <arg>--skip-dist-cache</arg>
        <arg>-m</arg>
        <arg>1</arg>