Move file from HDFS one directory to other directory in HDFS using OOZIE?

Question

I am trying to copy a file from HDFS one directory to other directory in HDFS, with the help of shell script as a part of oozie Job, but i am not able to copy it through oozie.

Can we copy file from HDFS one directory to other director in HDFS using oozie.

when i am running the oozie job, i am not any getting error.

it is showing status SUCCEEDED but file is not copying to destination directory.

oozie Files are below.

test.sh

#!/bin/bash
echo "listing files in the current directory, $PWD"
sudo hadoop fs -cp /user/cloudera/RAVIOOZIE/input/*  /user/cloudera/RAVIOOZIE/output/
ls  # list files

my workflow.xml is

<workflow-app name="RAMA" xmlns="uri:oozie:workflow:0.5">
<start to="shell-381c"/>
<kill name="Kill">
    <message>Action failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
</kill>
<action name="shell-381c">
    <shell xmlns="uri:oozie:shell-action:0.1">
        <job-tracker>${jobTracker}</job-tracker>
        <name-node>${nameNode}</name-node>
        <exec>/user/cloudera/test.sh</exec>
        <file>/user/cloudera/test.sh#test.sh</file>
          <capture-output/>
    </shell>
    <ok to="End"/>
    <error to="Kill"/>
</action>
<end name="End"/>

and my job.properties

oozie.use.system.libpath=True
security_enabled=False
dryrun=True
jobTracker=localhost:8032
nameNode=hdfs://quickstart.cloudera:8020
oozie.wf.application.path=${nameNode}/user/cloudera/test/

please help on this. why file is not copying to my destination director.

please let me know is there any thing i missed.

your workflow.xml and shell script looks fine. Your may have to carefully look for error - even if it says succeeded. Is your echo statement in shell script getting printed ? — sorabh
I am able to see echo statements in "jobhistory/logs" but there its not showing any error .it showing in log " Oozie Launcher, capturing output data: ======================= hello, cloudera. I wish to list some files of yours listing files in the current directory, /var/lib/hadoop-yarn/cache/yarn/nm-local-dir/usercache/cloudera/appcache/application_1470284226059_0022/container_1470284226059_00ror.i.e" — Sai
Your question is labeled "Move file" but your post is actually about copying files. If you really want to move these files, you should try the native Oozie HDFS Action oozie.apache.org/docs/4.2.0/… — Samson Scharfrichter

Dennis Jaheruddin Dennis Jaheruddin · Accepted Answer · 2016-08-10T09:31:01

As mentioned in the comments by @Samson:

If you want to do hadoop actions with oozie, you should use a hdfs action rather than a shell action for that.

I am not sure why you don't get an error, but here is some speculation on what might happen:

You give oozie the task of starting a shell action, it succesfully starts the shell action and reports a success. Then the shell action fails, but that's not oozies problem.

Move file from HDFS one directory to other directory in HDFS using OOZIE?

1 Answers