message get stucked when using message ttl to implement schedule message, RabbitMQ

0
votes

I need to schedule delay messages, so I declare a delay queue DELAY_STAT with x-dead-letter-exchange and x-dead-letter-routing-key which will route to the default exchange to my destination queue STAT.

When I need to schedule a delay message to queue STAT, I publish the message to DELAY_STAT with expiration, and when the message ttl are reached, it will be a dead letter and then be routed to STAT so that it can be consumed.

But I encounter a problem, according to https://www.rabbitmq.com/ttl.html#per-message-ttl-caveats

When setting per-message TTL expired messages can queue up behind non-expired ones until the latter are consumed or expired

which means some short delay message A will be stucked by long delay message B if B is sent earlier.

1
rabbitmqschedule

1 Answers

0
votes

If you want to schedule messages to deliver you may to do the following.

  1. When you are publishing a message to the queue you have to calculate a TTL for the message.
  2. In RabbitMQ you have to publish the message to the queue for the specific time window.
  3. When TTL expire the message will die and using x-dead-letter-exchange will transfer to destination queue where will be consumed.

E.g.

Let's assume that your time window is 60 seconds, so you will have a separate queue for messages with TTL between 0 and 59 seconds (inclusive) with name e.g. queue.dlx.0, a separate queue for messages with TTL between 60 and 119 seconds with name queue.dlx.0 and so on.

You know that for current message TTL equal to 50 seconds. Then, you may define delayed queue name queue.dlx + Math.floor(50 / 60)

When message TTL expires it dies and goes to queue where you have a consumer to process the messages.

Also, you may create a queue for each message and setup queue TTL (x-expires) to delete the queue after expiring message TTL (message TTL + several seconds just in case)

See this picture: https://hsto.org/getpro/habr/post_images/1a2/95d/14f/1a295d14fbeea180e17636a86e8fbce6.png

(source - https://habr.com/post/235983/ [RUS])

Also, you may take a look at this plugin https://github.com/rabbitmq/rabbitmq-delayed-message-exchange.

Math.floor - JS function to round fractional number to lowest integer number.