I have an app where each user is able to create tasks, and each task the user creates is added to a dynamic queue for the specific user. So all tasks from User1 are added to User1_queue, User2 to User2_queue, etc.
What I need to happen is when User1 adds Task1, Task2, and Task3 to their queue, Task1 is executed and Celery waits until it is finished before it executes Task2, and so on.
Having them execute along side each other from multiple queues is fine, so Task1 from both User1_queue, and Task1 from User2_queue. Its just limiting Celery to synchronously execute tasks in a queue in the order they're added.
Is it possible to have Celery have a concurrency of 1 per queue so that tasks are not executed alongside each other in the same queue?
If it helps anyone that visits this question, I solved my problem by setting up multiple workers with a concurrency of 1, each on a unique queue. I then used some logic in my Django app to store a queue name per session for the active user.
Down the line I'll add extra logic to select the 'least busy' worker, and try to evenly spread users across the active workers. But for now it is working perfectly.
Flower, the monitoring tool for Celery, was also a huge help while trying to figure this out.
