Bash Operator error: No such file or directory in airflow

14
votes

I am a newbie to Airflow and struggling with BashOperator. I want to access a shell script using bash operatory in my dag.py.

I checked: How to run bash script file in Airflow and BashOperator doen't run bash file apache airflow

on how to access shell script through bash operator.

This is what I did:

 cmd = "./myfirstdag/dag/lib/script.sh "

        t_1 = BashOperator(
            task_id='start',
            bash_command=cmd
        )

On running my recipe and checking in airflow I got the below error:

[2018-11-01 10:44:05,078] {bash_operator.py:77} INFO - /tmp/airflowtmp7VmPci/startUDmFWW: line 1: ./myfirstdag/dag/lib/script.sh: No such file or directory
[2018-11-01 10:44:05,082] {bash_operator.py:80} INFO - Command exited with return code 127
[2018-11-01 10:44:05,083] {models.py:1361} ERROR - Bash command failed

Not sure why this is happening. Any help would be appreciated.

Thanks !

EDIT NOTE: I assume that it's searching in some airflow tmp location rather than the path I provided. But how do I make it search for the right path.

5
pythonbashairflowdirected-acyclic-graphs
Is ./myfirstdag/dag/lib/script.sh relative to the $AIRFLOW_HOME/dags directory? - SergiyKolesnikov
@SergiyKolesnikov no it's not. /myfirstdag/dag/lib/ is a different path while $AIRFLOW_HOME gives a different path when I tried. - Marvin
what is then the absolute path to script.sh? - SergiyKolesnikov
@SergiyKolesnikov this /home/notebook/work/myfirstdag/dag/lib/ . I tried giving this too. It throws the same error. - Marvin
Apparently, it's searching in a tmp directory that it's creating. That's what I understood from the source code. github.com/apache/incubator-airflow/blob/… . Not sure how to make it search in the path I gave. - Marvin

5 Answers

7
votes

Try this:

bash_operator = BashOperator(
    task_id = 'task',
    bash_command = '${AIRFLOW_HOME}/myfirstdag/dag/lib/script.sh '
    dag = your_dag)
3
votes

For those running a docker version.

I had this same issue, took me a while to realise the problem, the behaviour can be different with docker. When the DAG is run it moves it tmp file, if you do not have airflow on docker this is on the same machine. with my the docker version it moves it to another container to run, which of course when it is run would not have the script file on.

check the task logs carefully, you show see this happen before the task is run. This may also depend on your airflow-docker setup.

1
votes

Try the following. It needs to have a full file path to your bash file.

cmd = "/home/notebook/work/myfirstdag/dag/lib/script.sh "

t_1 = BashOperator(
    task_id='start',
    bash_command=cmd
)
0
votes

Are you sure of the path you defined?

cmd = "./myfirstdag/dag/lib/script.sh "

With the heading . it means it is relative to the path where you execute your command.

Could you try this?

cmd = "find . -type f"
0
votes

try running this:

path = "/home/notebook/work/myfirstdag/dag/lib/script.sh"
copy_script_cmd = 'cp ' + path + ' .;'
execute_cmd = './script.sh'

t_1 = BashOperator(
    task_id='start',
    bash_command=copy_script_cmd + execute_cmd
)