rabbitmq round-robin consumption by celery workers

Question

I'm using a RabbitMQ broker, and there is a Celery worker which is subscribed to the broker. From my testing, it looks like RabbitMQ treats messages in FIFO order. Because one queue has been populated, then another, then another, and so on, my worker consumes all the messages from queue 1, and only moves on to queue 2 once it is done with queue 1.

Is it possible to change this behavior? I would like the Celery worker to consume in a round-robin style instead, ie consume a message from queue 1, then a message from queue 2, and so on, only coming back to queue 1 once a message has been consumed from each of the other queues.

Answer 1

Yes, you have to reduce your prefetch_count to 1 so only 1 message is fetched at a time. In Celery you can archive this by setting CELERYD_PREFETCH_MULTIPLIER to 1. You may also want to set task_acks_late = True, make sure you read the documentation on both.

from the Celery docs:

To disable prefetching, set worker_prefetch_multiplier to 1. Changing that setting to 0 will allow the worker to keep consuming as many messages as it wants.