RabbitMQ subsequent message atomicity

Question

I have got a RabbitMQ message structure, where a message A is supposed to generate a number of messages, let's call them B and C. A message A is received by a worker process, which then processes it and generates messages B and C.

The supposed workflow process is as follows:

1. Receive a message A with ack=False
2. Start a transaction
3. Run some code
4. Generate message B
5. Generate message C
6. Send ack for message A
7. Complete the transaction

In any case a worker process dies during the processing of message A, or while is has not yet completed the transaction - I would like RabbitMQ to treat the message A as undelivered and re-queue it.

RabbitMQ runs in highly available configuration if that is relevant.

Why question is trying to clear the RabbitMQ documentation here, stating:

Furthermore, RabbitMQ provides no atomicity guarantees even in case of transactions involving just a single queue, e.g. a fault during tx.commit can result in a sub-set of the transaction's publishes appearing in the queue after a broker restart.

• Is there any way I could achieve the behaviour I would like in the context of RabbitMQ or any other queueing software on the market?
• Is there any way I could make that work for multiple queues with RabbitMQ?

Answer 1

You are going to need quite a bit of infrastructure (code) to get this working just right. It is rather tricky.

You may want to consider a service bus. I have a free .net open-source one here: http://shuttle.github.io/shuttle-esb/

If you are not in the .net space or are not interested in using a service bus you could follow some of the code to see how some of these things have been handled by Shuttle.Esb: https://github.com/Shuttle/Shuttle.Esb

You may also want to take a look at the IdempotenceService here: https://github.com/Shuttle/Shuttle.Esb.SqlServer/blob/master/Shuttle.ESB.SqlServer/Idempotence/IdempotenceService.cs

When using the idempotence server (in a handler) all sent message are stored in a transaction store (like sql server in this case) and are only sent after the message handling has completed successfully.

For sending outside of a message handler one could use a transactional outbox (again, a sql server table-based queue would work).

The point is that you need to give your design some serious thought and it is going to be tricky.

Answer 2

I made a mistake by targeting the most popular product, as it seemed to me. ActiveMQ has full support for high-availability and transactions, so I have just switched to that and my problems went away.

Answer 3

We've got a few systems that work exactly like this in production, with that basic format of:

1. Get message A
2. Do work
3. Acknowledge message A

Where if step #2 fails, message A is "re-queued" and available to be picked up again.

When a message is sent to a queue and successfully received, that message is marked as ready. When it's then sent to a consumer it's marked as unacked until acknowledged by said consumer at which point it's removed from the queue.

In your example, until you've acknowledged the message in step #6 it's still in the unacked state, meaning if the worker process died before step #6 the message would slide back to the ready state in RabbitMQ, ready to be sent to another worker.

This functionality does rely on the fact that you're not explicitly disabling acknowledgements. If you were, I'm pretty sure messages are immediately removed from the queue upon being sent to a consumer.