0
votes

I'm getting this error when I submit a sqoop job using oozie Details below. Could you please help?

my workflow.xml file

<workflow-app xmlns="uri:oozie:workflow:0.4" name="oozie-wf">
<start to="sqoop-wf1"/>
<action name="sqoop-wf1">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<command>import --username hadoop --password bigdata --connect  
jdbc:db2://<host string>:50000/msrc --driver com.ibm.db2.jcc.DB2Driver --  
query  
'SELECT * FROM STUDENT.SALES WHERE (1=0 OR $CONDITIONS)' --target-dir  
/user/${wf:user()}/${examplesRoot}/apps/sqoop -m 1</command>
</sqoop>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed</message>
</kill>
<end name="end"/>
</workflow-app>

my job.properties file

nameNode=hdfs://sandbox.hortonworks.com:8020
jobTracker=sandbox.hortonworks.com:8050
queueName=default
examplesRoot=examples
oozie.use.system.libpath=true
oozie.libpath=/user/oozie/share/lib
oozie.wf.application.path=${nameNode}/user/root/${examplesRoot}/apps/sqoop

Invoking Sqoop command line now >>>

3638 [main] WARN org.apache.sqoop.tool.SqoopTool - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 2015-10-10 05:54:21,293 WARN [main] tool.SqoopTool (SqoopTool.java:loadPluginsFromConfDir(177)) - $SQOOP_CONF_DIR has not been set in the environment. Cannot check for additional configuration. 4540 [main] INFO org.apache.sqoop.Sqoop - Running Sqoop version: 1.4.6.2.3.0.0-2557 2015-10-10 05:54:22,195 INFO [main] sqoop.Sqoop (Sqoop.java:(92)) - Running Sqoop version: 1.4.6.2.3.0.0-2557 5581 [main] WARN org.apache.sqoop.tool.BaseSqoopTool - Setting your password on the command-line is insecure. Consider using -P instead. 2015-10-10 05:54:23,236 WARN [main] tool.BaseSqoopTool (BaseSqoopTool.java:applyCredentialsOptions(1021)) - Setting your password on the command-line is insecure. Consider using -P instead.

recognized argument: 15582 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Error parsing arguments for import: 2015-10-10 05:54:23,237 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(303)) - Error parsing arguments for import: 5582 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: * 2015-10-10 05:54:23,237 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: * 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: FROM 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: FROM 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: STUDENT.SALES 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: STUDENT.SALES 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: WHERE 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: WHERE 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: (1=0 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: (1=0 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: OR 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: OR 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: $CONDITIONS)' 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: $CONDITIONS)' 5583 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: --target-dir 2015-10-10 05:54:23,238 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: --target-dir 5584 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: /user/root/examples/apps/sqoop 2015-10-10 05:54:23,239 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: /user/root/examples/apps/sqoop 5584 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Unrecognized argument: -m 2015-10-10 05:54:23,239 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: -m 5584 [main] ERROR org.apache.sqoop.tool.BaseSqoopTool - Un 2015-10-10 05:54:23,239 ERROR [main] tool.BaseSqoopTool (BaseSqoopTool.java:hasUnrecognizedArgs(306)) - Unrecognized argument: 1 Intercepting System.exit(1)

<<< Invocation of Main class completed <<<

Failing Oozie Launcher, Main class [org.apache.oozie.action.hadoop.SqoopMain], exit code [1]

Oozie Launcher failed, finishing Hadoop job gracefully

Oozie Launcher, uploading action data to HDFS sequence file: hdfs://sandbox.hortonworks.com:8020/user/root/oozie-root/0000035-151009124402969-oozie-root-W/sqoop-wf1--sqoop/action-data.seq 2015-10-10 05:54:23,322 INFO [main] zlib.ZlibFactory (ZlibFactory.java:(49)) - Successfully loaded & initialized native-zlib library 2015-10-10 05:54:23,323 INFO [main] compress.CodecPool (CodecPool.java:getCompressor(153)) - Got brand-new compressor [.deflate]

Oozie Launcher ends

2015-10-10 05:54:23,346 INFO [main] mapred.Task (Task.java:done(1038)) - Task:attempt_1444199299654_0255_m_000000_0 is done. And is in the process of committing 2015-10-10 05:54:23,426 INFO [main] mapred.Task (Task.java:commit(1199)) - Task attempt_1444199299654_0255_m_000000_0 is allowed to commit now 2015-10-10 05:54:23,438 INFO [main] output.FileOutputCommitter (FileOutputCommitter.java:commitTask(482)) - Saved output of task 'attempt_1444199299654_0255_m_000000_0' to hdfs://sandbox.hortonworks.com:8020/user/root/oozie-root/0000035-151009124402969-oozie-root-W/sqoop-wf1--sqoop/output/_temporary/1/task_1444199299654_0255_m_000000 2015-10-10 05:54:23,509 INFO [main] mapred.Task (Task.java:sendDone(1158)) - Task 'attempt_1444199299654_0255_m_000000_0' done.

2

2 Answers

0
votes

As the sqoop is trying to parse the whole command using Java, and it will parse with spaces like mentioned below, you can try like below mentioned and it will work fine for you.

Try this,

<workflow-app xmlns="uri:oozie:workflow:0.4" name="oozie-wf">
<start to="sqoop-wf1"/>
<action name="sqoop-wf1">
<sqoop xmlns="uri:oozie:sqoop-action:0.2">
<job-tracker>${jobTracker}</job-tracker>
<name-node>${nameNode}</name-node>
<arg>import</arg>
<arg>--connect</arg>
<arg>jdbc:db2://<host string>:50000/msrc</arg>
<arg>--username</arg>
<arg>hadoop</arg>
<arg>--password</arg>
<arg>bigdata</arg>
<arg>--driver</arg>
<arg>com.ibm.db2.jcc.DB2Driver</arg>
<arg>--query</arg>
<arg>SELECT * FROM STUDENT.SALES WHERE (1=0 OR $CONDITIONS)</arg>
<arg>--target-dir</arg>
<arg>/user/${wf:user()}/${examplesRoot}/apps/sqoop</arg>
<arg>-m</arg>
<arg>1</arg>
</sqoop>
<ok to="end"/>
<error to="kill"/>
</action>
<kill name="kill">
<message>Action failed</message>
</kill>
<end name="end"/>
</workflow-app>
-1
votes

When using the command element, Oozie will split the command on every space into multiple arguments.Passing --arg is fine if you've a simple sql query but if you've a complex query with multiple joins and where conditions passing multiple --arg can be tedious,in which case you should create sqoop jobs and execute them from oozie.

1.configure sqoop-site.xml to use sqoop metastore change following properties

sqoop.metastore.client.autoconnect.url,sqoop.metastore.client.autoconnect.username,sqoop.metastore.client.enable.autoconnect,sqoop.metastore.client.autoconnect.password

https://sqoop.apache.org/docs/1.4.2/SqoopUserGuide.html#_literal_sqoop_metastore_literal

2.Start sqoop metastore

sqoop metastore & 

3.Create sqoop jobs

sqoop job --create jobid -- import ...

3.Submit sqoop job in oozie using

job --exec jobid --meta-connect sqoop-metastore-URL