RabbitMQ - Consume multiple queues

4
votes

I am working with Python, RabbitMQ and Pika.

I have multiple consumers and queues. I want one consumer to get messages from every queues. Let's say I have 2 consumers and 3 queues. I want each of my consumers to read messages from queue number 1, 2 and 3.

I have manage to do so with a basic_get :

basic_get(queue1)

basic_get(queue2)

basic_get(queue3)

=> Get one message from queue 1, one from queue 2, one from queue 3, then repeat.

The "problem" is that I want to use a basic_consume in order to set qos (let queues to push n messages each time). I want to get n messages from queue 1, then n messages from queue 2, n messages from queue 3 and then go back to queue 1 and so on. I don't want to consume all from queue 1, then all from the 2nd, ...

I have not find a way to implement basic_consume for multiple queues. Is it possible to implement basic_consume in my situation?

Also, do I need threads when using basic_consume? It does call a function when a message is sent in RabbitMQ. But this is a listener as I understand. So I cannot do anything while it is listening. Using threads would be helpful to process tasks while listening to new messages, right?

Thanks.

1
pythonrabbitmqamqppika
Please note it is not thread safe to share Pika connections across threads - each thread must have its own connection: pika.readthedocs.io/en/latest/faq.html – Robben_Ford_Fan_boy

1 Answers

0
votes

set the prefetch for the consumer, and you should be able to do that.

I'm not sure about the Pika code specifically, but the example in that documentation shows it like this:

channel.basicQos(10); // Per consumer limit
channel.basicConsume("my-queue", false, consumer);

regarding threads... that depends on your specific application needs. often times, it is easier to increase the # of process instances instead of using threads. but using threads certainly is a valid way to parallelize the processing of the messages in a single process.