Issues running Airflow DAG

Question

I have been working on Apache Airflow for a while now to schedule my workflow. I seem to be having issues scheduling my DAG. I have been using this SO question for reference : Airflow not scheduling Correctly Python

from airflow import DAG 
from airflow.operators.bash_operator import BashOperator
from datetime import datetime
from datetime import timedelta

default_args = {
    'owner': 'airflow',
    'depends_on_past': False,
'start_date': datetime.now() - timedelta(minutes=5),
'email': ['[email protected]'],
'email_on_failure': False,
'email_on_retry': False,
'retries': 1,
'retry_delay': timedelta(minutes=5),
}

 dag = DAG('dag_mkdir_folder', default_args=default_args,
      schedule_interval=timedelta(minutes=5))


 task_hello = BashOperator(task_id='print_hello',
                      bash_command='mkdir test_airflow', dag=dag)

I'm trying to run the task using the following list of commands :

 airflow scheduler
 airflow trigger_dag dag_mkdir_folder

I keep getting this as an error :

[2017-05-15 13:49:06,688] {models.py:322} DagFileProcessor406 INFO -      Finding 'running' jobs without a recent heartbeat
[2017-05-15 13:49:06,689] {models.py:328} DagFileProcessor406 INFO -   Failing jobs without heartbeat after 2017-05-15 13:44:06.689284

The bash command is just supposed to create a new directory. The test version works fine.

Answer 1

Run the scheduler on a different terminal and then trigger your dag in another terminal

also try providing your full path where you want to make the directory. For example creating folder in the airflow directory:

task_hello = BashOperator(task_id='print_hello',
                  bash_command="mkdir ~/airflow/test_airflow", dag=dag)

This should create test_airflow folder inside airflow

Answer 2

Your current bash_command is telling Airflow to create a directory inside of the temporary directory that the DAG uses when it is running, which it blows away along with all of its contents after the DAG is run.

Consider changing directories to the directory you want to create it in.

A bash_command with that would look something like:

"cd <path_to_directory>; mkdir test_airflow"