Azure Service Bus, AWS SNS, RabbitMQ -> All subscribers get the message?

Question

While looking at the Pub/Sub pattern, i came across the fellowing scenario:

Assume that you have a horizontally scaled app, that has X instances. All of them subscribe to a topic where messages like "Transfer $10 from account A to account B". When someone publish a message to that topic, all subscriber will get that message?

In the case above, clearly, the message should be taken by only 1 subscriber and handled only once.

How does one handle this scenario? Do you abandon the pub/sub and starts pooling?

Answer 1

Let me explain few things with example before you understand that completely. I have worked on Azure service bus so i will explain in that context.

In Pub/sub you have one topic and possible multiple subscription. Lets say we have topic "Shopping-Topic". We have 2 Subscriptions called "Payment-Subscription", "Cart-Subscription". Now we publish message "Payment-processed" on the topic. It's the discretion of subscription to pick that message and reason is that subscription have to mention that which messages it want pick.

In Azure service bus we have something called rule (message label). Default rule is that subscription is listening to all the messages but we can overwrite this behavior and say i am only interested in particular message. In the above case rule added against "Payment-Subscription" to listen the message "Payment-processed" so the message is added to "Payment-Subscription" subscription for it to process. Even though "Cart-Subscription" is also subscribed to the same topic but it is ignoring this message so it's not added to its subscription. This way any intended subscription can listen to particular message not necessarily all of them.

Now we discuss individual subscription. Let's say we have message added to "Payment-Subscription". This subscription has 2 instances/processes that are ready to process the message "Payment-processed". The first process to pick the message will process the message and remove it from subscription.

Answer 2

In RabbitMQ Normally, active consumers connected to the same queue receive messages from it in a round-robin fashion. So this insures that a message is processed exactly once.

So in your case you should design a queue where all the messages for

"Transfer $10 from account A to account B"

Are routed to and all the consumers register themselves on this queue itself , this insures that one message will go to only one subscriber.

Another point not related to your question but is important to know is that there is another concept called "Consumer Priorities" which allows you to ensure that high priority consumers receive messages while they are active, with messages only going to lower priority consumers when the high priority consumers block.

More info can be found here