Django-celery - How to execute tasks serially?

Question

I'm using Celery 3.1. I need to only execute the next task when the last one is finish. How can I assure that there are not two tasks working at the same time? I've read the documentation but it is not clear for me.

I've the following scheme:

Task Main
    - Subtask 1
        - Subtask 2

I need that when I call "Task Main" the process will run till the end(Subtask 2) without any new "Task Main" starting.

How can I assure this?

Answer 1

One strategy is through the use of locks. The Celery Task Cookbook has an example at http://docs.celeryproject.org/en/latest/tutorials/task-cookbook.html.

Answer 2

If I understand you want to execute only MainTask one by one, and you want to call subtasks in your MainTask. Without creating separate queues and at least 2 separate workers this is impossible. Because if you will store in same queue all tasks looks for celery as same tasks.

So solution for is:

1. map MainTask to main_queue
2. Start separate worker for this queue like: celeryd --concurrency=1 --queue=main_queue
3. map subtasks to sub_queue
4. Start separate worker for this queue celeryd --queue=sub_queue

Should work!

But I think this is complecated architecture, may be you can make it much easier if you will redesign your process.

Also you can find this useful (it works for you but it could run parallel MainTask): You should try to use chains, here is an example on Celery's docs: http://docs.celeryproject.org/en/latest/userguide/tasks.html#avoid-launching-synchronous-subtasks.