Airflow BashOperator: Passing parameter to external bash script

Question

Having problems passing parameters to an external bash script from a BashOperator. When I run a local command, the params are substituted correctly:

log_cleanup = """ echo "{{ params.BASE_LOG_FOLDER }}" """
log_cleanup_task = BashOperator(
        task_id='log_cleanup_task',
        provide_context=True,
        bash_command = log_cleanup,
        params = {'BASE_LOG_FOLDER': "/var/opt"},
        dag=dagInstance,
)

prints:  "/var/opt"   (without the double quotes)

But if I call an external bash script, the params don't substitute in.

log_cleanup_task = BashOperator(
        task_id='log_cleanup_task',
        provide_context=True,
        bash_command= str(DAGS_FOLDER)+"/scripts/log_cleanup.sh ",
        params = {'BASE_LOG_FOLDER': "/var/opt" },
        dag=dagInstance,
)

#log_cleanup.sh:
#! /usr/bin/bash
echo "{{ params.BASE_LOG_FOLDER }}"


prints: "{{ params.BASE_LOG_FOLDER }}"    (without the double quotes)

In the external bash script, I can't get the parameters to substitute in like they do when the statement is stored within the DAG .py script.

Do I have to pass the params as command line arguments instead? Does the jinja templating only works in the .py files?

Can you please try the following in your Bash Script: echo {{ params.BASE_LOG_FOLDER }} i.e by removing quotes around it — kaxil
[2020-02-25 21:10:36,904] {bash_operator.py:128} INFO - {{ params.BASE_LOG_FOLDER }} — wolf2600

kaxil kaxil · Accepted Answer · 2020-02-25T21:13:47

Remove the space after "log_cleanup.sh " in bash_command

So your task should become:

log_cleanup_task = BashOperator(
        task_id='log_cleanup_task',
        provide_context=True,
        bash_command= "scripts/log_cleanup.sh",
        params = {'BASE_LOG_FOLDER': "/var/opt" },
        dag=dagInstance,
)

Note that the scripts folder should be inside the folder containing your DAG file and it should contain the relative path to script (relative to folder containing this DAG)

The main reason you got TemplateNotFound error was the path mentioned in bash_command is not recognized by Jinja (templating engine used by Airflow). Jinja only recognizes path passed in DAG.template_searchpath The default path is the folder containing the DAG so you can directly place your scripts folder under DAGs folder if your DAG is directly in the $AIRFLOW_HOME/dags. Or you can pass the path to your folder in DAG.template_searchpath as follows:

dag = DAG("example_dag", template_searchpath="/var/opt/scripts")

# And then just pass "filename" to bash_command
log_cleanup_task = BashOperator(
        task_id='log_cleanup_task',
        provide_context=True,
        bash_command= "log_cleanup.sh ",
        params = {'BASE_LOG_FOLDER': "/var/opt" },
        dag=dag,
)

Airflow BashOperator: Passing parameter to external bash script

1 Answers