Azure Service Bus Topic - Handling duplicate messages in topic queue

0
votes
  • We have an azure topic with duplicate message detection property enabled.
  • Because of the above setting, any messages with message ID similar to the previous message for a certain duration of time gets ignored by the azure webjob function.
  • However, we would like to investigate and know more information about the messages that gets ignored. Is there any option available for us to run a logic for duplicate messages ?
2
azure-functionsazure-servicebus-topics
Good question, but I think you can't. P.S. You are probably asking about Azure Service Bus Topic, not Azure Storage queue. I would change the title and tags of the question if so. – Mikhail Shilkov

2 Answers

0
votes

Because of the above setting, any messages with message ID similar to the previous message for a certain duration of time gets ignored by the azure webjob function.

Azure Service Bus duplicate detection is based on message ID over defined time window. It is Azure Service Bus broker that de-duplicates, not Azure Webjob/Function. De-duplicated messages never pass the broker point.

If messages are de-duplicated, that's a clear sign it was a duplicate. If you don't trust that, I suggest either to change your consumer logic to implement idempotency and stop relying on Service Bus to de-duplicate message for you or change how you generate message IDs. The idea is to create message ID based on the message payload.

0
votes

  1. Enabling duplicate detection helps keep track of the application-controlled MessageId of all messages sent into a queue or topic during a specified time window. If any new message is sent carrying a MessageId that has already been logged during the time window, the message is reported as accepted (the send operation succeeds), but the newly sent message is instantly ignored and dropped. No other parts of the message other than the MessageId are considered. (the blog referenced in one of the responses says the message content cannot be duplicate which is not correct).

  2. Default value of duplicate detection time history now is 30 seconds, the value can range between 20 seconds and 7 days.

Alternatively, you can go with tools like Serverless360, Here this will have an option to regenerate the messageid.

Refer this blog for more details