Messages stacking on consumers, when other consumers are available in RabbitMQ - using bunny for rails

Question

Queue messages in RabbitMQ are stacking and waiting on a single consumer while other consumers are available.

We use RabbitMQ as our messenger service. We use bunny to create connections and setup queues with passing messages.

In our rails setup using bunny, we are having an issue where we have one queue with 8 consumers listening on that queue for messages. When messages come in, ideally they should be round robin-ing the consumers example: 4 messages in the queue, consumer 1 picks up message 1, - consumer 1 is busy, consumer 2 picks up message 2, consumer 2 is busy, consumer 3 picks up message 3, and so forth.

However the issue we are having is, 4 messages in the queue, consumer 1 picks up message 1, consumer 1 busy, consumer 2-8 are available, but messages 2-4 are stacked in the queue waiting on consumer 1 to become available and process the messages.

I feel like i've done a bunch of research and just can't quite figure out how to stop messages from stacking and waiting on a single consumer.

Has anyone had experience in this, or have any ideas on how to fix this issue?

conn = Bunny.new(bunny[0])
conn.start
ch = conn.create_channel
q = ch.queue("#{record_queue_name}", :durable => true)
q.subscribe(:manual_ack => true, :arguments => {"x-priority" => 10}, :block => true) do |delivery_info, properties, payload|
ch.acknowledge(delivery_info.delivery_tag, false)

We are wanting anytime any messages get sent to RabbitMQ, the consumer picks them up on a first come first serve basis, not multiple messages stacked waiting for a busy consumer when others are available.

EDIT: How to reproduce: start 3 consumers at the same time. push through 6 messages - consumers 1 - 3 are now busy with 3 messages in the queue. restart 2 & 3, when 2 & 3 are listening again, 3 messages are still waiting in queue waiting on consumer 1. consumer 2 & 3 are still available.

Restart consumer 1, now 3 queue'd messages first come first server to newly restarted consumers 2 & 3.

I need messages to first come first server regardless if consumers restart.

It is weird to be passing an x- argument via subscribe. If you provide code that reproduces this reliably in a repository that I can clone and run I can investigate. — Luke Bakken
I do not have any additional code that I can share publicly. - Also I did want to add, if all consumers are created at the same time and are listening it does it on a first come first serve, however if any of the consumers are restarted while theres messages in the queue waiting, they will not grab waiting messages once they are back up and listening, this is initially what causes the stacking of waiting messages. — joe
You've added a very important detail in your comment. Please edit your original question and give exact steps on how to reproduce this. If you can't share a minimal, runnable code example to reproduce the issue it is very unlikely I will spend the time to guess on how to address it. — Luke Bakken
Original updated with how i duplicate the issue, the code logic we have that a consumer handles allows it to remain busy enough to have messages waiting in the queue for consumers. — joe

Luke Bakken Luke Bakken · Accepted Answer · 2019-07-31T13:05:24

RabbitMQ is working as intended.

Since your code does not set QoS / prefetch, RabbitMQ sends all six messages to your first consumer. Since that consumer takes time to acknowledge messages (simulated by a 45 second sleep in the code), those six remain in the "Unacked" state while your other two consumers don't have anything to work on. Restarting those other two consumers has no effect since all six messages are in "Unacked" waiting for an ack from the first consumer.

When you restart your first consumer, RabbitMQ detects the lost connection and enqueues the six messages to the "Ready" state, and delivers all six (most likely) to another consumer, and the problem repeats.

Please see this runnable code sample for how to set prefetch. With a prefetch of 1, RabbitMQ will at most deliver one message to a consumer and will wait for an ack before delivering another message to that consumer. In that way, messages will be distributed among your consumers.

_{NOTE: the RabbitMQ team monitors the rabbitmq-users mailing list and only sometimes answers questions on StackOverflow.}

Messages stacking on consumers, when other consumers are available in RabbitMQ - using bunny for rails

1 Answers