RabbitMQ - Topic Exchanges and Competing Consumers

2
votes

I successfully setup a Topic Exchange, and I'm able to deliver messages to several consumers at once.

I'd also like to deliver messages to competing consumers and keep using topic exchanges. I read that using the same queue name enables consumers to compete for messages. I might be mistaken, however, since I cannot make it work.

Setup for several listeners to the same topic:

  • Declare topic exchange
  • For each listener, declare a new queue with an autogenerated name
  • Bind this queue to the above exchange with a given topic routing key

How to setup competing consumers to the same topic?

Is it even possible with Topic Exchanges?

Thanks.

2
c#.netrabbitmq

2 Answers

6
votes

Let's review a couple of points first.

First, remember that in RabbitMQ you always consume from queues. Exchanges are just your portals and you cannot directly consume from them.

Second, Topic exchanges allow for binding the queues with routing key "patterns". Therefore, the term topic is valid in the context of "Topic Exchanges".

Now this is what I understand from your question:

Multiple consumers/same routing key: This is where you want multiple consumers to all consume the messages with the same routing key (or same routing key patterns in the case of Topic Exchanges). This is in fact doable. Just do this:

  1. Declare your Topic Exchange
  2. Declare a queue with some name
  3. Bind that queue to your topic with your desired routing key pattern
  4. Create multiple consumers and have them listen to that same queue.

What will happen is that RabbitMQ is going to load balance to your consumers in a round robin matter. This means all consumers will consume from the same queue. But remember that in this scenario it is possible that a single message is delivered more than once in theory.

What you were doing was to create multiple queues and have one consumer per queue. This means that every message coming to the exchange would be duplicated across all queues. The end result would be that a message gets processed multiple time.

2
votes

I solved this by using exchange-to-exchange bindings.

  1. The outer exchange is a topic exchange.
  2. The inner exchange is a fanout exchange bound to a client-named queue.
  3. The outer exchange is bound to the inner exchange with a routing key that includes a wildcard.