can we parameterize the airflow schedule_interval dynamically reading from the variables instead of passing as the cron expression

2
votes

can we parameterize the airflow schedule_interval dynamically reading from the airflow variables instead of passing directly as the cron expression

I have passed the following way as per airflow documentation args = { 'owner': 'pavan', 'depends_on_past': False, 'start_date': datetime(2020, 1, 15), 'email_on_failure': True, 'email_on_retry': False, 'retries': 0, 'on_failure_callback': notify_email }

with DAG(dag_id=DAG_NAME, default_args=args, schedule_interval='* 1 * * *', catchup=False) as dag:

1
airflowairflow-schedulerairflow-operator

1 Answers

1
votes

Yes

Technically you can do it, but it brings 2 problems

  • minor problem: reading a Variable means a SQL-query being fired to Airflow's SQLAlchemy backend meta-db. And doing it in your DAG-definition script means this will happen as the DAG is continuously parsed by Airflow in background. Read point 2 here

  • major problem: A Variable can be edited via UI. But altering an Airflow DAG's schedule_interval can have wierd behaviours and may require you to either rename the DAG or (anecdotal finding) restart the scheduler to fix that