RabbitMQ design advice: Routing DLX message back to originating publisher

1
votes

EDIT: The underlying requirement is that I have multiple web-service server processes receiving requests from web-service clients, performing some processing (what I want to use a worker queue for), and then returning the result back to to the respective client. However I want to be able to wait for a consumer of this queue to pick up a published request, within a timeout. If this timeout expires, then I want the original webservice process to receive a notification of this so it can feed back to the client.

  • I have multiple publishers publishing messages to the same exchange
  • There are multiple consumers on this exchange consuming from the same queue, i.e. 'task-queues'

I'm trying to implement a sort of 'Publish TTL', whereby the originating publisher (only) is notified if a message is dead-lettered. (I want the publisher to know if a message is not consumed within a certain timeout).

I have a solution as follows:

  • When publishing messages, each publisher uses a routing key containing a unique (per publisher) Process-ID, e.g. 'thekey.pid3' for publisher 3
  • Consumers each consume from the same queue, that is bound to the exchange using the routing key 'thekey.#'
  • I'm not explicitly specifying any routing-key for the DLX when publishers declare the queue -- only the 'x-dead-letter-exchange'. This means that if a message is deadlettered, then the original routing-key (i.e. 'thekey.pid3') will carry through to the DLX.
  • Each publisher binds it's own new consumer queue to the DLX using it's specific routing key, e.g. 'thekey.pid3'

So this resolves my requirement, as only the publisher that published the message will have the DLX message routed to it.

This, however, seems a little convoluted -- and inefficient, as the wildcard routing occurs for ALL messages (except those dead-lettered -- which shouldn't normally be happening).

This must be a common requirement -- is my solution advisable? Is there a more efficient way to do this? Or am I on the right track?

Here's a diagram showing my design:

EDIT: In the below diagram, the DLX should actually be connected to the queue, not the Exchange! DLX back to originating publisher

Thanks!

1
rabbitmqamqp
It really depends on your needs, SO may not be the best choice for discussions, try RabbitMQ official user group – pinepain
Thanks Zaq. I've edited the question with a brief overview of the underlying BL requirement in case that clarifies things. However I'll post this on the Rabbit group too. – Ive
RabbitMQ mailing list discussion about this question is here: groups.google.com/d/msg/rabbitmq-users/xVsSV5R1sk8/DxpN-sbYhBIJ – old_sound

1 Answers

0
votes

Right, so it turns out the solution is quite valid. There may be a small latency hit when using a wildcard binding key and topic exchange, but probably nothing to really concern oneself about.

One comment from the mailing list was that because this is an RPC-based system, each Publisher already has a unique response-queue defined, so one could bind that queue to the DLX (rather than creating a new, unique-queue-per-publisher), and thereby get both 'valid' responses and dead-lettered messages delivered to the same queue.

In my case, I wanted to make wrap this 'Publish TTL' functionality in a generic publisher, so referring to the response-queue didn't make sense -- although my solution may be slightly less efficient, as there is one more round-trip in the event of a dead-letter.

If you're interested in the mechanism the guys at Rabbit use for topic routing, you can check out: http://www.rabbitmq.com/blog/2010/09/14/very-fast-and-scalable-topic-routing-part-1/ http://www.rabbitmq.com/blog/2011/03/28/very-fast-and-scalable-topic-routing-part-2/