Celery Executor & tasks sequencing

Question

I am using airflow with celery executor. As an example, I have a dag with 3 tasks. task1 >> task2 >> task3

When my worker nodes = 1, I see that the tasks execute fine and in the right sequence

However, when I increase the worker nodes to 2, I see that task1 is picked by worker node 1 and task2 by worker node 2. In my use case, task2 must execute only after task1. I believed that the celery executor will understand this. My dag fails because of this

Can you please me understand what’s wrong and how do I fix this issue?

Answer 1

The simplest solution is to use the Chain primitive. The whole point of Celery as a distributed task queue system is to run tasks concurrently on different nodes, so do not be alarmed when your tasks are executed on different machines.