Airflow: Only allow one instance of task

Question

Is there a way specify that a task can only run once concurrently? So in the tree above where DAG concurrency is 4, Airflow will start task 4 instead of a second instance of task 2?

This DAG is a little special because there is no order between the tasks. These tasks are independent but related in purpose and therefore kept in one DAG so as to new create an excessive number of single task DAGs.

max_active_runs is 2 and dag_concurrency is 4. I would like it start all 4 tasks and only start a task in next if same task in previous run is done.

Answer 1

From the official docs for trigger rules:

depends_on_past (boolean) when set to True, keeps a task from getting triggered if the previous schedule for the task hasn’t succeeded.

So the future DAGs will wait for the previous ones to finish successfully before executing.

Answer 2

I may have mis-understood your question, but I believe you are wanting to have all the tasks in a single dagrun finish before the tasks begin in the next dagrun. So a DAG will only execute once the previous execution is complete.

If that is the case, you can make use of the max_active_runs parameter of the dag to limit how many running concurrent instances of a DAG there are allowed to be.

More information here (refer to the last dotpoint): https://airflow.apache.org/faq.html#why-isn-t-my-task-getting-scheduled

max_active_runs defines how many running concurrent instances of a DAG there are allowed to be.

Answer 3

Airflow operator documentation describes argument task_concurrency. Just set it to one.

Answer 4

On airflow.cfg under [core]. You will find

dag_concurrency = 16 //The number of task instances allowed to run concurrently by the scheduler

you're free to change this to what you desire.