Use priority with a RabbitMQ DLX queue

2
votes

I have 2 RabbitMQ queues:

  • incoming_message => where I push all messages that I want to process later
  • incoming_message_dlx => where I push the message whose the processing failed

As you can supposed with its name, the incoming_message_dlx queue use the Dead Letter Exchange feature, that means when the message expires, it will be requeue to my incoming_message.

What I try to achieve is to increase the expiration of messages each time the processing failed and that they are push to the DLX queue.

The problem is that even if a message expired, it will not be requeue to my incoming_message while it's not at the bottom (head) of the queue. So if there is a message with an expiration time of 7 days in the DLX queue and that we enqueue a new message with the expiration time of 5 seconds, this message will only be requeue to the incoming_message after 7 days + 5 seconds...

I've found on the documentation that I can use my DLX queue as a priority queue and put a priority on my messages according to the expiration time, but it doesn't work as expected, the priority seems to be ignored.

However, when I use the RabbitMQ admin (management plugin) and that I get the first message of the queue, it's always the one with the higher priority, but the "internal consumer" of the DLX queue seems to ignore this priority.

Do you know what could be the problem?

Thanks a lot in advance.

PS: I'm using RabbitMQ server version 3.6.10.

1
rabbitmqqueuepriority-queue

1 Answers

0
votes
  1. as a queue structure(fifo),rabbitmq do expire from the head of the queue.
  2. queue ttl contains 3 type:
    • Per-Queue Message TTL： x-message-ttl
    • Per-Message TTL： expiration
    • Queue TTL：x-expires

when you want the message just deliver on the ttl value ,try use multi level ttl queue.

you can predefined some dlx queue as you need.

eg: you want error message do retry in (5s,15s,60s), you can define 3 dlx queue by set different x-message-ttl value, and this 3 incoming_message_dlx queue binding the dlx router to the incoming_message;

but if you have a message ttl=30s ,you just prefdefind 3 queue with ttl(5s,15s,60s) , so where to diliver ? try priority queue.

offical doc

Messages which should expire will still only expire from the head of the queue. This means that unlike with normal queues, even per-queue TTL can lead to expired lower-priority messages getting stuck behind non-expired higher priority ones. These messages will never be delivered, but they will appear in queue statistics.

expired lower-priority messages getting stuck behind non-expired higher priority ones

queue like [60s(p=1),30s(p=0)] will not happen!

we defined 3 queue ttl(5s,15s,60s),in order to prevent lower ttl message getting stucked , we push the message to the queue with flor ttl not ceil ttl；

so ttl=30s will deliver to queue which ttl=60s,and set priority=1

  • ttl=30s is between the predefined queue (15s,60s);
  • set ttl=60s queue's max-priority=1, default is 0;
  • deliver ttl=30s message with priority=1;

so the message in a queue just like [30,60,60,60,60].
ttl=30s will not be blocked by the ttl=60s.