Supporting manual and scheduled execution

Question

My organization has been using a DAG that only runs on a manual trigger for quite some time. Any external resources that the dag interacts with are parameterized by the execution date {{ ds_nodash }}.

We recently converted this dag to run on a weekly schedule and I came to find that the airflow "scheduler triggers a DAG run at the end of its schedule period, rather than at the beginning of it". I wasn't expecting this at all. I'm not trying to debate airflow's schedulers design, but instead I'm looking for some advice on how to write DAGs that work both by manual trigger AND by a scheduled interval.

Here are my specific dilemmas:

1. To get around the fact that the scheduler uses the beginning of the period I could use {{ next_ds_nodash }}, but then the manually triggered dag will never be parameterized by its own execution date
2. When manually triggered we wanted to look at another file 7 days in the past: {{ execution_date - macros.timedelta(days=7)).strftime("%Y%m%d") }}. When schedule triggered this now looks 14 days in the past. How can I accomplish both?

I'm not looking for direct solutions to my dilemmas, but instead wondering if my intention of having a DAG run on a schedule and manually is actually quite out of the ordinary? If not what are specific solutions to my dilemmas OR some best practices for writing DAGs that you can manually trigger AND schedule.

Answer 1

Every time I had the need to run a DAG both on a schedule and occasionally in a manual way, I always... created two DAGs!

I think this is actually the official recommendation by Airflow itself. Just make two DAGs and set one with a weekly schedule like schedule_interval = '0 5 * * 1' # Every Monday at 5:00 and one with schedule_interval = None. Give them IDs like my_awesome_dag and my_awesome_dag_manual.

With this setup you can play around and make sure it always look at the file that is relevant for that execution_date. You can then keep them both ON in the GUI and while the weekly one will trigger indeed every week, the other one will trigger only when you press the play button.