Issue with Airflow version 1.10.1

Question

Recently I have upgraded airflow version to 1.10.1. I turned on some of the dags turned on which were marked OFF earlier. I am always using start_date for dag as today's date. After turning ON the dags it brought below issue.

Scheduler is however starting those DAG's but it is not picking related task's. On task Instance Detail its showing "The execution date is 2018-12-04T13:00:00+00:00 but this is before the task's start date 2019-02-04T00:00:00+00:00." It runs only after triggering it manually.

Is there any other way (apart from fixing the start_date for DAG) this issue can be fixed. i.e; using some config or any other option where I can by-pass the above check of execution date and task's start date.

My main purpose is to run dag's old schedule without manual intervention.

Answer 1

You should not use dynamic start date especially not today's date or datetime.now(). Have a read on official docs https://airflow.readthedocs.io/en/stable/faq.html#what-s-the-deal-with-start-date for more details.

I know you asked for the suggestion apart from start date but your start date definitely needs to be before the task execution date. Hence, I would strongly recommend changing your start_date to something like datetime(2018, 1, 1).