RabbitMQ Work-Queue for FanOut messaging

Question

I'm trying to scale-out a RabbitMQ messaging system. The current system is very simple - the producer sends a message to a fanout exchange and the message is handle by multiple consumers - classic fanout routing .

I have multiple consumers from different types (e.g: one that print to screen, one that logs to file, one that saves to DB,...). My challenge - i'm not sure what's the best way to scale-out the consumers. If i add other consumers from the same type - i'll get double logs or double entries in the DB. ... (think about two DB consumers consuming from the same fanout exchange) .

I guess I can create a consumer that publish to a work-queue but I wonder if there's a better "builtin" solution in rabbitmq.

thanks in advance, zf

Answer 1

If you need to scale consumers in order to consume faster all the messages coming from the fanout exchange you need competive consuming; so you need more consumers attached to the same queue bound to the fanout exchange.
In this way every consumer will consume a batch of messages indipendently from the others. The number of messages inside the batch is defined with the prefetch count property ( http://www.rabbitmq.com/consumer-prefetch.html ).
In this way, in your case, you should be able to scale consumers avoiding double logs and double entries in the DB.