Dynamically created dags get deleted automatically from Airflow UI

Question

I am quite new to Apache Airflow so I am not sure if all features of airflow.

My requirement is to create the dynamic dags for the entries which are newly added to database. The criteria for checking if anything is newly added to DB is if the dag_id is updated against that entry or not. If not then we need to create a new dag for it and update the dag_id against that entry.

New entry in database with Dag_id as Null

Id  :d001   ,
Dag_id :Null

Once main dag (which is built to create dynamic dags) runs and fetches this entry, a dag is created and dag_id is updated back to database.

Id : d001 ,
Dag_id : dagid-001

The challenge I am facing is once the main dags runs again for fetching newly added entries (this time we won't be retrieving d001). The dynamic dags(dagid-001) which were previously created are getting deleted from Airflow UI and the Airflow list_dags as well.

code from main dag to create dynamic dag

dag_id = 'hello_world_{}'.format(str(new_entry[0]))

    default_args = {'owner': 'airflow',
                    'start_date': datetime(2018, 1, 1)
                    }

    schedule = ## frequency 

    dag_number = new_entry

    globals()[dag_id] = create_dag(dag_id,
                                  schedule,
                                  dag_number,
                                  default_args)

Is this the expected behavior of dynamic dag or am I missing something while creating them?

SergiyKolesnikov SergiyKolesnikov · Accepted Answer · 2020-06-29T20:16:26

For Airflow, a DAG exists only it the corresponding DAG object is found while scanning Python files in the dag directory.

So, your main DAG has to generate DAGs not only for new DB entries, but also for all previous entries too.

Dynamically created dags get deleted automatically from Airflow UI

1 Answers