RabbitMQ multiple consumers across multiple queues - messages delayed from being processed

1
votes

We have recently experienced unexpected behaviour with our application that is powered by RabbitMQ. RabbitMQ version is 3.6.12 and we are using .NET Client 5.0.1

The application subscribes to two queues, one for commands, and another for events - we also use manual acknowledgements. Our application is configured to have 7 consumers. Each has its own channel(IModel) and each has its own EventingBasicConsumer We end up processing messages when EventingBasicConsumer.Received is fired.

Our application must process messages as close as possible to when they are routed onto the queues and to date we have not had issues. However recently, we have seen that when one of our messages being processed takes a long time to complete, it delays when another message is to be processed although there are many consumers available (6) that are not busy.

Note we have observed that this issue does not happen when an application is only subscribing to a single queue, it becomes an issue when there is multiple queues involved.

This is best illustrated using the following example:

  • We have a simple consuming application that subscribes to two queues, one for commands and one for events. This application have 7 consumers, each with their own channel and EventingBasicConsumer We start a simple publishing application, that publishes 20 messages, a second apart. Every message is an event so is published to the event queue except for the 5th and 10th messages, which are commands and sent to the command queue. Note that every event is processed without delay whereas commands take 30 seconds

  • The following table describes what we are observing in relation to assigning multiple channels to messages across multiple queues:

enter image description here

  • Once Message5 completes after 30 seconds with C1, then Messaqe9 is assigned immediately to C1 and is processed without delay Once Message10 completes after 30 seconds with C2, then Messaqe11 is assigned immediately to C2 and is processed without delay

  • Hence, to us it looks like the assignment of channels is done independently per queue - meaning you can have delayed execution if some messages take a long time to process.

Is it possible that when multiple consumers are subscribing to multiple queues, RabbitMQ can assign a message to be handled by a consumer that is busy even if there are consumers that are currently idle?

Is there any documentation that explains the RabbitMQ algorithm that selects which consumers EventingBasicConsumer.received fires from a collection of consumers?

1
multithreadingrabbitmqmessaging

1 Answers

3
votes

We have fixed this issue.

In the RMQ documentation (https://www.rabbitmq.com/api-guide.html#consuming) we came across the following: "Each Channel has its own dispatch thread. For the most common use case of one Consumer per Channel, this means Consumers do not hold up other Consumers. If you have multiple Consumers per Channel be aware that a long-running Consumer may hold up dispatch of callbacks to other Consumers on that Channel.”

In our code, we had 2 consumers per channel, meaning consumers could hold up other consumers. We changed to have one consumer per channel and that fixed the issue.