best practices with poison message handling for Azure service bus topic

Question

Dealing with poison messages (throwing exception while consuming) from Azure Service Bus can lead to loops till number of retries has reached maxDeliveryCount setting of topic subscription.

Does the SequenceNumber of message added by Azure Service bus keeps on increasing on each failed attempt till it reaches maxDeliveryCount ?
Setting maxDeliveryCount = 1, is that best practice to deal with poison messages so that consumer never attempt twice to process message once it failed

Hi @Fortm, there might be scenarios where you need to keep an eye on the faulty messages that come into your Topic Subscription and the dead-lettered messages of your topic subscription. One big challenge could be the Repair and Resubmitting of those messages. Nevertheless, that challenge can be overcome using tools like Service Bus Explorer or Serverless360. — Nadeem Duke
is challenge with Resubmitting related to the fact that resubmitted message goes to end of queue and it effectively inverts order ? — fortm

Pankaj Rawat Pankaj Rawat · Accepted Answer · 2020-04-12T13:52:34

Best practices depend on your application and your retry approach.

Most of time I noticed message get failed

Dependent service not available (Redis, SQL connection issue)
Faulty message (message doesn't have a mandatory parameter or some value is incorrect)
Process code issue (bug in message processing code)

For the 1st and 3rd scenario, I created C# web job to run and reprocess deadletter message.

Below is my code

internal class Program
    {
        private static string connectionString = ConfigurationSettings.AppSettings["GroupAssetConnection"];
        private static string topicName = ConfigurationSettings.AppSettings["GroupAssetTopic"];
        private static string subscriptionName = ConfigurationSettings.AppSettings["GroupAssetSubscription"];
        private static string databaseEndPoint = ConfigurationSettings.AppSettings["DatabaseEndPoint"];
        private static string databaseKey = ConfigurationSettings.AppSettings["DatabaseKey"];
        private static string deadLetterQueuePath = "/$DeadLetterQueue";

        private static void Main(string[] args)
        {

            try
            {
                ReadDLQMessages(groupAssetSyncService, log);
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.Message);
                throw;
            }
            finally
            {
                documentClient.Dispose();
            }
            Console.WriteLine("All message read successfully from Deadletter queue");
            Console.ReadLine();
        }

        public static void ReadDLQMessages(IGroupAssetSyncService groupSyncService, ILog log)
        {
            int counter = 1;
            SubscriptionClient subscriptionClient = SubscriptionClient.CreateFromConnectionString(connectionString, topicName, subscriptionName + deadLetterQueuePath);
            while (true)
            {
                BrokeredMessage bmessgage = subscriptionClient.Receive(TimeSpan.FromMilliseconds(500));
                if (bmessgage != null)
                {
                    string message = new StreamReader(bmessgage.GetBody<Stream>(), Encoding.UTF8).ReadToEnd();
                    syncService.UpdateDataAsync(message).GetAwaiter().GetResult();
                    Console.WriteLine($"{counter} message Received");
                    counter++;
                    bmessgage.Complete();
                }
                else
                {
                    break;
                }
            }

            subscriptionClient.Close();
        }
    }

For 2nd scenario, we manually verify deadletter messages (Custom UI/ Service Bus explore), sometimes we correct message data or sometimes we purge message and clear queue.

I won't recommend maxDeliveryCount=1. If some network/connection issue occurs, the built-in retry will process and clear from the queue. When I was working in a finance application, I was keeping maxDeliveryCount=5 while in my IoT application is maxDeliveryCount=3.

If you are reading messages in batch, a complete batch will re-process if an error occurred any of message.

SequenceNumber The sequence number can be trusted as a unique identifier since it is assigned by a central and neutral authority and not by clients. It also represents the true order of arrival, and is more precise than a time stamp as an order criterion, because time stamps may not have a high enough resolution at extreme message rates and may be subject to (however minimal) clock skew in situations where the broker ownership transitions between nodes.

best practices with poison message handling for Azure service bus topic

1 Answers