Azure service bus dropping messages?

Question

I am trying to build a simple Web API REST service in Azure with a service bus queue worker on the back end. I can send a single message from the Web API to the worker just fine. However, I was trying to send more messages just to see how everything works. So, I created a simple controller that looks like this:

for (int i = 0; i < 100; i++)
{
    var msg = new BrokeredMessage("Ping");
    BioConnector.QueueConnector.OrdersQueueClient.Send(msg);
}

When I call the controller, I am only getting about 1/2 or so of the messages being received by the worker. The rest seem to be dropped.

Answer 1

I had issues with getting only about half the messages using the sample code posted here, so I wrote my own test code. I've tried it with > 100 queue messages and have always had 100% send/rec'd parity. Perhaps you had a similar issue with the code.

1. Create a new C# console project.
2. Add a reference to the Microsoft.ServiceBus assembly located in C:\Program Files\Microsoft SDKs\Windows Azure.NET SDK\2012-06\ref\Microsoft.ServiceBus.dll.

3. In the app.config, change it to this with your own values provided:

   <appSettings>
       <add key="Microsoft.ServiceBus.ConnectionString" value="Endpoint=sb://blahblah.servicebus.windows.net/;SharedSecretIssuer=owner;SharedSecretValue=pDk0b....=" />
   </appSettings>
   

4. Add these using directives:

   using System;
   using System.Collections.Generic;
   using System.Linq;
   using System.Text;
   using System.Threading.Tasks;
   using System.Xml;
   using System.Xml.Linq;
   using Microsoft.ServiceBus;
   using Microsoft.ServiceBus.Messaging;
   using System.Configuration;
   using System.Threading;
   

5. Change the code method to the following:

   class Program
   {
       static void Main(string[] args)
       {
           string connectionString = ConfigurationSettings.AppSettings["Microsoft.ServiceBus.ConnectionString"];
           var namespaceManager = NamespaceManager.CreateFromConnectionString(connectionString);
           QueueDescription queueDesc = new QueueDescription("TestQueue");
           if (!namespaceManager.QueueExists(queueDesc.Path))
           {
               namespaceManager.CreateQueue(queueDesc);
           }
           QueueClient topicClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
           int sentMsgCount = 0;
           int recdMsgCount = 0;
   
           for (int i = 0; i < 100; i++)
           {
               BrokeredMessage msg = new BrokeredMessage("Test message " + i);
               topicClient.Send(msg);
               sentMsgCount++;
               Console.WriteLine("Sent Message: " + msg);
           }
   
           QueueClient subClient = QueueClient.CreateFromConnectionString(connectionString, queueDesc.Path);
   
           bool moreMessages = true;
           while (moreMessages)
           {
               BrokeredMessage recdMsg = subClient.Receive(TimeSpan.FromSeconds(3));
               if (recdMsg != null)
               {
                   Console.WriteLine("Received Message: " + recdMsg);
                   recdMsgCount++;
                   recdMsg.Complete();
               }
               else
               {
                   moreMessages = false;
               }
           }
           Console.WriteLine("# of sent msgs: " + sentMsgCount + ", # of rec'd msgs: " + recdMsgCount);
           Console.Read();
       }
   }
   

Answer 2

This was a weird problem. Through a random walk through "trying things" I ended up changing the string name of the queue and then everything started working again. I didn't change anything but the name of the queue - no changes to any configuration parameters at all.

It appears to be something buggy with that particular queue on Azure.

Answer 3

Azure Service Bus provides durable messaging so you will not loose any messages. Some items to investigate further: 1) Is there another instance of the worker role that is pulling off messages from that queue 2) are you using peek-lock as the receive mode as that will be the only way to guarantee at least once delivery. Receive and delete mode does not have the guarantee 3) are the messages going into dead-lettered queue due to either message expiry or exceeding max delivery count, I.e. They are received but not completed several times 4) if none of the above apply then raise a support ticket and the Azure product team can investigate the symptoms because as I mentioned this is a durable messaging system so no messages will be "lost".

Answer 4

I had the WindowsAzure.ServiceBus NuGet package in my project and used QueueClient.Send() to send messages and faced the same messages lost issue.

My solution to completely solve the issue:

On the send side, I had to use REST API to send messages.

On the receive side, this is how I extracted the message body:

using (var stream = brokeredMessage.GetBody<Stream>())
{
    using (var streamReader = new StreamReader(stream, Encoding.UTF8))
    {
        var msg = streamReader.ReadToEnd();
        // Convert the JSON message to an object
        // var obj = JsonConvert.DeserializeObject<ObjectType>(msg);
    }
}