How do I implement competing consumers of consumers that can consume multiple things?

0
votes

I'd like to implement competing consumers using RabbitMQ, I have 2 (or more) consumers:

Consumer 1 can consume messages of type "red" and of type "blue"
Consumer 2 can consume messages of type "red" and of type "black"

Now I'd like to have load balancing between the consumers (and possibly more instances of consumers with the same configurations) using the competing consumer pattern and I'd like to keep message ordering "loosely" intact (I don't need perfect ordering but would want a message from 2 hours ago to be processed earlier than one from 2 minutes ago).

Example: I'd like to process the following messages in the following way:

  • Blue: Consumer 1
  • Black: Consumer 2
  • Red: The consumer that finished processing first (assume Consumer 1)
  • 10 million red messages: Both consumers will compete for the messages and process them
  • 1 million blue messages: only after all 10 million red are done consumer 1 should start to process the blue messages
  • 1 million red messages: only after consumer 1 is done with the blue messages it should start processing the new red messages, since no black messages are there for consumer 2 it can immediatly begin processing the red messages.

Is there any way to achieve this using RabbitMQ? If I just make a queue for each message type I see no way to ensure message ordering.

If RabbitMQ is not the right tool for the job is there maybe something more suitable?

1
rabbitmqmessaging

1 Answers

0
votes

You can solve this using a priority queue. By giving higher priority to the red messages, your consumers will fetch the red messages first. When no more red messages are left, your consumers will start consuming the blue messages.

While using priority queue, make sure you keep a lower basic.qos value. This allows the messages to be prioritised in the queue.