When consumer gets message from channel in rabbitmq,where does pre-fetch messages reside

Question

I have below configuration for rabbitmq

prefetchCount:1 ack-mode:auto.

I have one exchange and one queue is attached to that exchange and one consumer is attached to that queue. As per my understanding below steps will be happening if queue has multiple messages.

1. Queue write data on a channel.
2. As ack-mode is auto,as soon as queue writes message on channel,message is removed from queue.
3. Message comes to consumer,consumer start performing on that data.
4. As Queue has got acknowledgement for previous message.Queue writes next data on Channel.

Now,my doubt is,Suppose consumer is not finished with previous data yet.What will happen with that next data queue has written in channel?

Also,suppose prefetchCount is 10 and I have just once consumer attached to queue,where these 10 messages will reside?

Answer 1

The scenario you have described is one that is mentioned in the documentation for RabbitMQ, and elaborated in this blog post. Specifically, if you set a sufficiently large prefetch count, and have a relatively small publish rate, your RabbitMQ server turns into a fancy network switch. When acknowledgement mode is set to automatic, prefetch limiting is effectively disabled, as there are never unacknowledged messages. With automatic acknowledgement, the message is acknowledged as soon as it is delivered. This is the same as having an arbitrarily large prefetch count.

With prefetch >1, the messages are stored within a buffer in the client library. The exact data structure will depend upon the client library used, but to my knowledge, all implementations store the messages in RAM. Further, with automatic acknowledgements, you have no way of knowing when a specific consumer actually read and processed a message.

So, there are a few takeaways here:

1. Prefetch limit is irrelevant with automatic acknowledgements, as there are never any unacknowledged messages, thus
2. Automatic acknowledgements don't make much sense when using a consumer
3. Sufficiently-large prefetch when auto-ack is off, or any use of autoack = on will result in the message broker not doing any queuing, and instead doing routing only.

Now, here's a little bit of expert opinion. I find the whole notion of a message broker that "pushes" messages out to be a little backwards, and for this very reason- it's difficult to configure properly, and it is unclear what the benefit is. A queue system is a natural fit for a pull-based system. The processor can ask the broker for the next message when it is done processing the current message. This approach will ensure that load is balanced naturally and the messages don't get lost when processors disconnect or get knocked out.

Therefore, my recommendation is to drop the use of consumers altogether and switch over to using basic.get.