i am going to put specific file to hdfs with shell script that run on oozie workflow. i have error when run shell script and my job is killed

Question

when i run workflow on oozie that contain shell script i have error. yarn application log show error:

>>> Invoking Shell command line now >>

Exit code of the Shell command 1
<<< Invocation of Shell command completed <<<

i am going to download file from ftp server an put file to hdfs path with oozie workflow. i define workflow with shell script in shell script i download file with wget and then use hdfs dfs -put command to load file from local to hdfs

when run shell script on master node it work correctly. this is shell script: (sample.sh)

wget ftp://ftpserver/test.txt
hdfs dfs -put test.txt hdfs://master1:8020/user/ambari-qa

i have tow problem: 1- when i run wget alone. my job in oozie finished succeeded state. but i do not know where test.txt is located??? (which path saved test.txt). i try this command to define path for wget ftp://ftpserver/test.txt -P /home/ambari-qa but job is killed when i use /home/ambari-qa local path in script. 2- when i use hdfs dfs -put alone, job is killed. i do not know when i use hdfs dfs command in my script then my job is killed. consider my shell script have problem when i use that in oozie and i have to change that. please advise correct command in my shell script

oozie workflow.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<workflow-app xmlns="uri:oozie:workflow:0.5" name="test">
    <start to="shell_1"/>
    <action name="shell_1">
        <shell xmlns="uri:oozie:shell-action:0.3">
            <job-tracker>${resourceManager}</job-tracker>
            <name-node>${nameNode}</name-node>
            <exec>/user/ambari-qa/sample.sh</exec>
            <file>/user/ambari-qa/sample.sh</file>
        </shell>
        <ok to="end"/>
        <error to="kill"/>
    </action>
    <kill name="kill">
        <message>${wf:errorMessage(wf:lastErrorNode())}</message>
    </kill>
    <end name="end"/>
</workflow-app>

Your <file> tells Oozie to download hdfs:///user/ambari-qa/sample.sh to the local filesystem of the YARN container, without renaming it. Therefore you should simply <exec>sample.sh</exec> — Samson Scharfrichter
Looks like you really need a tutorial on Oozie basics. Try the "Hooked on Hadoop" 13-parts series; a bit old but still the best around.. — Samson Scharfrichter
i used <exec>sample.sh</exec> but my result did not change — ahmad
@vikrant -- the shell script file must be on hdfs and after run workflow the file downloaded on datanode in yarn container and run container on datanode — ahmad

Dmitry Dmitry · Accepted Answer · 2021-05-18T11:30:31

This error is not informative in case you have invalid shell script that results with code different from 0. Considering this you should check first correct EOL or other behaviour of your bash that can cause problem and then try with oozie.

i am going to put specific file to hdfs with shell script that run on oozie workflow. i have error when run shell script and my job is killed

1 Answers