DAG executes OK from web interface, but “falls through” when run via command line

Question

I am trying to test my DAG using command line interface, during development of new functions, but I am not able to do that. My DAGs, DAG_ID=sample_dag, file: sample_dag.py resides in ~/airflow/dags folder (Ubuntu) and can be executed OK via Web Interface (by clicking on Play icon). There are a few BASH OPERATOR calls within DAG and each script is executed correctly, producing logged output.

However, I cannot access the same DAG's functionality via command line, ran from the same folder, e.g.: airflow render sample_dag all 2019-01-14T06:04:05

The output of the above command is: Test Dag Begin Test Dag End ****airflow.exceptions.AirflowException: dag_id could not be found: sample_dag. Either the dag did not exist or it failed to parse.****

The Debug prints within DAG are executed, but no Bash script is called (no script log is produced).

import datetime as dt
from airflow import DAG
from airflow.operators.bash_operator import BashOperator

default_args = {
    'owner': 'airflow',
    'start_date': dt.datetime(2019, 1, 12),
    'retries': 1,
    'retry_delay': dt.timedelta(minutes=5),
    'email_on_failure': False,
    'email_on_retry': False,
}

dag = DAG('sample_dag', default_args=default_args, schedule_interval=None)

t0 = BashOperator(
    task_id='all', bash_command='echo "Starting Tests..."', dag=dag)

t1 = BashOperator(
    task_id='test_1',
    bash_command='python3 /home/ubuntu/scripts/test_1.py &>> /home/ubuntu/scripts/test.log',
    dag=dag)

t2 = BashOperator(
    task_id='test_2',
    bash_command='python3 /home/ubuntu/scripts/test_2.py &>> /home/ubuntu/scripts/test.log',
    dag=dag)

t3 = BashOperator(
    task_id='final', bash_command='echo "Client Profile Tests Complete"', dag=dag)

print("Test Dag Begin")

t0 >> t1
t0 >> t2
t1 >> t3
t2 >> t3

print("Test Dag End")

Yes, it is... Message is very confusing, as the Dag runs in a way - print commands are actually executed with airflow render or airflow run. However, no Bash Operator code gets called from command line, only from web interface. I suspected issues with python interpreter getting called via Bash Operator, but it is not. The same behavior happens if Bash commands are just: echo TestN &>> /home/ubuntu/scripts/test.log. Really do not understand what is going on. Help would be appreciated. — Imios Archangelis
I also tried to run: airflow backfill -s -1 sample_dag [2019-01-15 15:02:23,685] {jobs.py:2212} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2 [2019-01-15 15:02:28,692] {jobs.py:2212} INFO - [backfill progress] | finished run 0 of 1 | tasks waiting: 3 | succeeded: 1 | running: 0 | failed: 0 | skipped: 0 | deadlocked: 0 | not ready: 2... Getting more confused: how tasks just executing echo cannot be ready?! — Imios Archangelis
Huh, the only command which actually works as expected seems to be: airflow trigger_dag sample_dag. It seems that "airflow render" just displays "what is to be executed", but doesn't execute anything (very different from e.g. graphics rendering). What "airflow run" is supposed to do, I do not understand at all. — Imios Archangelis
From a quick look, this answer of mine could apply to your case as well: stackoverflow.com/a/52610828/825190 — Alessandro Cosentino

kaxil kaxil · Accepted Answer · 2019-01-15T20:41:28

The airflow render cli command just renders the parameters of an operator that are templated. It won't run your DAG.

airflow render [-h] [-sd SUBDIR] dag_id task_id execution_date

You should use trigger_dag to run your dag using cli

DAG executes OK from web interface, but “falls through” when run via command line

1 Answers