Why does Airflow keep running the DAG?

0
votes

I am learning Airflow for a Data Engineering project, and I setup a DAG to retrieve a csv file online. I was testing out the schedule_interval and I set it to 30 mins initially.

I started the Airflow scheduler at 22:17, and expecting the DAG to be executed at least at 22:47. However, the DAG is running almost at every second, and I see from the log that the execution date was a few hours ago.

DAG

Is this because of the time difference from UTC to my local time? The DAG is trying to catch up to the time difference?

2
pythonairflowairflow-scheduler

2 Answers

0
votes

Your DAG is being backfilled. Airflow will attempt to catch up to your current time from when it was started.

E.g. if the exact moment in which you launched your DAG is on 6th March, 10:00AM, and the DAG has an execution date of 6th March 6:00AM (assuming the same timezone), with a scheduling interval of 30 mins, then the DAG will run immediately until it has "caught up" to 10:00AM.

That is, it would run (6:00AM - 10:00AM = 4 hours; 4 hrs/30 mins = 8) 8 times one after another until it has reached the current moment in time.

Is this because of the time difference from UTC to my local time? The DAG is trying to catch up to the time difference?

Seems like it, if the DAG's execution start date is whatever time you launched your DAG at.

0
votes

It would be very helpful. If you can paste the DAG as well or atleast the DAG configuration object.

Make sure you set the flag catchup=False so that backfilling does not happen. The default value is True. If you did not set catchup=False scheduler assumes that it needs to backfill and hence it is running every 30secs.

See the example below

dag = DAG(
    dag_id='my_test_dag'
    , default_args=default_args
    , schedule_interval='1 * * * *'
    , start_date=datetime(2020, 9, 22, tzinfo=local_tz)
    , catchup=False
)