Prioritise celery queues/tasks

Question

I have the basic understanding of celery and that how it works. In my current project, I have run into a need to prioritise tasks. I mean if there are two kinds of tasks, say A and B in the celery queue, celery should prioritise task B irrespective of which task is on the head of the queue. Is there a way to do that?

Queue prioritisation is also fine with me. Meaning that I can make 2 different queues, say high_priority_queue and low_priority_queue, and celery should always execute the tasks in high_priority_queue first and then go towards low_priority_queue.

I also know the fact that we can assign different workers to the two queues, but that would mean that tasks in both the queues are being executed concurrently. I need the tasks in the high_priority_queue to be executed first. Any ideas?

Thanks

Answer 1

Usually the mutliple worker approach with multiple queues is recommended but as you pointed out, the low priority queue/worker will be working concurrently next to the high priority worker. This is an interesting setup if you have a lot of small tasks that you want to have executed rather soon, thus you put them in the high priority queue while the longer tasks get pushed to the low priority queue. You could also have a setup where you simply give more resources (or a better machine) to the high priority worker.

Since you would like a different solution I am going to suggest the priority parameter for apply_async. You do need a bit of setup for that as pointed out in a different question I answered recently and it only works for certain brokers. (For RabbitMQ it works since version 3.5.0.) After having set the x-max-priority on your queue and the additional settings as pointed out in the referenced answer you can simply put a priority on a task like this:

your_task.apply_async(queue="your_queue_that_can_handle_priority", priority=10)