Azure Service Bus Topic: Why is message duplication caused?

Question

I have been using Azure Service Bus Topics in a production application. My understanding is the system guarantees at least one copy of the message on the subscription end when published.

In practice, I generally see 1 message corresponding to a publish. In some degenerate cases though I see multiples. For example: 1 publish that corresponds to ~20 messages in the relevant subscription. Currently nothing is consuming this subscription, which leads me to believe this is the publishing logic.

Under what circumstances are messages duplicated rampantly? I would imagine networking disturbances might cause this. What else?

Note that the default duplicate handling logic can't handle this. You would have multiple messages with the same body and distinct message IDs - which I understand is used to establish message "identity".

EDIT: My publishing logic is pretty uncomplicated and resembles the following:

public void PublishSomeId(Guid someId, int eventDelayInMinutes, string componentName)
        {
            var topicClient = ServiceBusManager.GetTopicClient();

            var message = new BrokeredMessage(someId);
            message.To = componentName;
            message.ScheduledEnqueueTimeUtc = DateTime.UtcNow.AddMinutes(eventDelayInMinutes);
            topicClient.Send(message);
        }

More details would be required to answer this question. What's your publishing logic? If you publish the same payload 20 times with a different ID, then yes, you'll have 20 "copies" of the event. What your topology looks like? Without this information answering this is very much guessing. — Sean Feldman
I have edited the question to add a code sample that closely resembles my actual implementation. Anecdotally I see ~20 messages with body 'someId'. With the help of logs, I know this method has only been invoked once with 'someId'. — Arxo Clay
Indeed, nothing fancy. What's your setup of topic and subsides under that topic? — Sean Feldman
I get that. Something in your setup is different though. Sender is dead simple. So is receiver. Topology is missing. Need to see it. Ping me at my email lastname.firstname at gmail. — Sean Feldman

Sean Feldman Sean Feldman · Accepted Answer · 2017-01-25T01:52:07

Two suggestions:

Verify that your publisher is not duplicating the messages. Add a log statement in PublishSomeId to see what's the someId value is. I suspect this method could be invoked multiple times. In addition to that, if you don't see duplicates logged, you could enable duplicates detection. For that, you should assign someId to the brokered message MessageId. Not sure what's the period of time that duplicates are generated, but you could experiment.
If #1 is not helping, it could the the topology setup (entities). Check there's nothing funky like fun out with fan in back into the same topic.

Azure Service Bus Topic: Why is message duplication caused?

1 Answers