RabbitMQ - Consume multiple queues by priority

6
votes

I have a requirement to upgrade our Producer/Consumer infrastructure.

The current setup look like this:

  • Set of 3 queues with different priority (low, med, high).
  • When one our customers generate a task (i.e Process an image):
    • The Producer add the message to the relevant queue.
    • One of the workers address it.

The issue with such approach, is that in case and a customer generate a huge amount of tasks, it can occupy all available slots in queue, which potentially can lead to deny of service (or a huge delay) in that queue.

Suggested changes:

  • Each customer (or a group of) should have dedicated Consumer (or a group of).
  • When the Consumers are idle, they should process other customers' messages.

For example, We have a set of messages:

 1. Producer: Customer1, Queue: High, Payload: {}, Created: Today 16:00:00
 2. Producer: Customer2, Queue: High, Payload: {}, Created: Today 16:00:01
 3. Producer: Customer1, Queue: High, Payload: {}, Created: Today 16:00:02
 4. Producer: Customer1, Queue: High, Payload: {}, Created: Today 16:00:03

And we have the following Consumers:

1. Consumer1: Dedicated for Customer1
2. Consumer2: Dedicated for Customer1
3. Consumer3: Dedicated for Customer2

Expected result:

1. Consumer1 will address Message#1
2. Consumer2 will address Message#2
3. Consumer3 will address Message#3
4. Message#4 Any of the Consumers can address it, since Consumer1/3 are dedicated to the given Producer and Consumer2 will be idle.

To sum things up, a customer should always get a dedicated amount of Consumers (or more when available) ASAP, whenever there is nothing to do, his Consumers can consume other messages of other customers.

I'm trying to figure out what is the best approach to achieve that goal even on the expense of switching from RabbitMQ to any other messaging broker.

The only approach I've found (Using RabbitMQ) so far is to use federate queues and to form a complete bi-directional graph (each queue is a upstream of all other queues and vise versa).

1
rabbitmqamqppriority-queueconsumerproducer

1 Answers

1
votes

There is no formula that would produce fair queuing for every type of workload; How much, how uniform and how long does it take to process tasks is important. That said, I doubt federated queues would be of any help regarding fairness.

You can work with priority queues and consumer priorities. In combination with low prefetch counts, it could be possible to attain a scheduling that suits your customer's expectations.

However, none of this mechanisms will throttle your customers based on their used capacity; If a customer sends too many high-priority slow-to-process tasks in short time, it will still block your other customers.

Since you are suggesting that one consumer per customer is a possibility, I understand that the number of customers is not a very large one. A solution to consider would be having per-customer prioritized queues. Messages from all these queues could be picked in a round-robin fashion by a buffer queue that is consumed by multiple workers, as explained in this diagram from a related question on the SE:

queing

This way, no customer gets an edge, and you somewhat decouple what fairness means (in your context) from the actual load balancing.

I have not tried myself but I think the shovel plugin could be sufficient to implement the re-queuing (orange in the diagram). If you want anything more elaborated, like quotas deciding what is fair and what is not, implementing your own re-queuing service could be worth the effort.