How to set per-queue memory limits for RabbitMQ?

2
votes

I have 3 applications A, B and C which use RabbitMQ, where I have queues Q1 and Q2.

  1. Application A publishes messages in Q1.

  2. Application B consumes messages from Q1, does some processing, and publishes other messages in Q2. B only acks the message from Q1 after it sucessfully publishes a message in Q2.

  3. Application C consumes messages from Q2, does some processing and writes something into a database.

If application A publishes faster than B can consume, it will eventually fill the memory or disc of the RabbitMQ server, which will block any clients that try to publish messages.

B is then blocked from publishing to Q2 before it can ack the message from Q1, leading to a deadlock.

If I could limit the memory and disc usage for each queue, rather than for the whole RabbitMQ, I think this could be solved. Is there a way to do this, or another way to solve my problem?

I could switch to another message broker that supported something like this if needed.

1
rabbitmqmessage-queue

1 Answers

1
votes

No, setting per-queue disc or memory limit is not what can be done in RabbitMQ (at least for now).

If messages can't be lost and are mission-critical, making application B consume with the same speed as A or faster, say, by increase consumers number may be the solution.

Another approach, is to drop extra messages, say, via Per-Queue Message TTL or with Queue Length Limit.

Sure, that messages can be dead-lettered for alarm notification and/or backup.

You can also check queue length time to time and do some heuristics to monitor that and correlate your workflow (note, queue.declare will return messages number in current queue but this AMQP method is idempotent by itself).