Currently, I have DAGs where schedule interval are defined in the DAG itself. I think I can use airflow variables for this also. Instead of variables I thought of creating a database table to store the values and use PostgresHook(or other db hooks) to pull data from table to populate the schedule_interval, start_date or default_args of DAGs. So just wondering if this is efficient/possible. I just thought that this might result to large unnecessary queries on the airflow database.
0
votes
Can you elaborate as to why this is something you're looking to do?
- joebeeson
The values for default_args, schedule_interval, etc is something that we want to be managed by users of different env (prod, stage, qa, or dev). So having a database table to do that might be feasible but I think this will be inefficient and might result to airflow db connection issues.
- alltej
1 Answers
1
votes
Changing schedule_interval can be hairy, especially if you have sensors that check for prior run success or wait_for_downstream set; anything that introspects the state of the database by execution_date really.
It could work and you could likely piggy-back on the models.Variable table but you'll have to deal with the increased database load, increased load time for your webserver, the scheduler will be slower and now doing any local testing/development requires that you now sync the databases.
We use a version controlled repository with multiple branches to keep code separated which also helps to enforce a peer review of code prior to being pushed to Airflow.
