In RabbitMQ which is more expensive, multiple queues per exchange, or multiple exchanges and less queues per each?

Question

So we decided to go with RabbitMQ as a message/event bus in our migration to micro-services architecture, but we couldn't find a definite answer on what is the best way of putting our queues, we have two options to go with:

1. One main exchange which will be a Fanout exchange, which in turn will fan messages out to a main queue for logging and other purposes and another sub exchange which will be a topic exchange and route the messages to each desired queue using the message routing key. We expect the number of queues behind the sub-exchange to be some how a large number. This can be explained by this graph:

2. One main exchange, which will be a Topic exchange, with still one main queue bound to that exchange using "#" routing key. That main exchange will also handles main routing to other sub exchanges, so routing keys might be "agreements.#", "assignments.#", "messages.#", which are then used to bind multiple topic sub-exchanges, each will handle sub routing, so one sub exchange might be handling all "assignments" and queues bound to that exchange could be bound by routing keys like "assignments.accepted", "assignments.deleted"...In this scenario, we feel like the huge number of queues will be less per exchange, they will be somehow distributed between exchanges. So, which of these scenarios could be the best approach? Faster on RabbitMQ, less overhead.

Taking in mind, all queues, exchanges and bindings will be done on the fly from the service which will be either publishing or subscribing.

Answer 1

You can find some explanation in this topic: RabbitMQ Topic exchanges: 1 Exchange vs Many Exchanges

I am using RabbitMQ in a very similar way that you showed in the case 2, as I found the same benefits as described in this article: https://skillachie.com/2014/06/27/rabbitmq-exchange-to-exchange-bindings-ampq/

Exchange-to-exchange bindings are much more flexible in terms of the topology that you can design, promotes decoupling & reduce binding churn

Exchange-to-exchange bindings are said to be very light weight and as a result help to increase performance *

Based in my own experience with exchange-to-exchange, the case 2 is great and it will allows to create/change messages flow topologies in a very fast way.

Answer 2

I'm going to first re-summarize what I think is your question, since I'm sure it's buried somewhere in your post.

It is desirable to have a tracer/logging queue, in addition to a series of work-specific queues for actual message processing. What exchange topology is best for this scenario?

First off, neither option makes much sense given your application. Option 1 will create an exchange that will publish a message to every queue bound to it, regardless. This is clearly not what you want. Option 2 will give you a rather complex routing topology for which the benefit is unclear, and the drawback is painful maintenance and a steep learning curve. (Just because you can do something does not mean you should do it.)

What should be done?

It is important to remember that in RabbitMQ, it is the queues which consume the resources of the broker. Exchanges merely connect queues with publishers. The exchange is a means to an end, while the queue is the end itself.

What instead I think you should do is set up a single topic exchange. Bind your tracing queue to routing key # so that you receive all messages. Then, bind your work queues appropriately so that they receive only the messages that need to flow into them. For example, it is common to route messages by message type, where each queue holds exactly one type of message. This is both simple and effective.

The advantage of a single topic exchange is that you get the benefits of both a Direct Exchange and a Fanout Exchange depending on the binding key used. Further, configuration changes are easy to achieve and can often be done without disrupting any system processing at all (let's say that you want to stop tracing certain messages - this can be done with ease using a topic exchange, assuming your routing keys are rational).