Service Bus Queue nightmare. Messages disappeared

2
votes

We are running a cloud service on two different​ instances. This cloud service spins up a receiver on a service bus queue. This receiver is configured as: 1. Peek and lock mode 2. Auto complete = true 3. RenewLockTimeout = 10 mins 4. MaxConcurrentMessages = 1 5. PrefetchCount = default (should be 0)

For what I know this means that a maximum of two messages are processed at the same time (one per instance) and messages are automatically removed from the queue as soon as the message handler is completed.

Now, few days ago Azure decided to update our instances. I've seen from the log that at 18.18 the first instance was requested​ to stop. At 18.24 the instance was stopped and at 18.27 came back online. At 18.28 the second instance was requested to stop and at 18.35 it was back online.

Seven messages has been enqueued at 18.23. One of them produced the expected record on the DB (i.e. the handler was executed successfully). The other 6, NO. Dead letter queue is enabled, but empty.

In theory my code should stop receiving messages as soon as the roles enter in the OnStop method. A cancellation token is canceled and the message handler is "paused" introducing a delay for 10 mins (so, for what I know, the message shouldn't be completed and then it should be processed again). So I cannot say If the first message has been processed by the first or the second machine. But it was processed. And I'm sure that the other 6 were in the queue.

If there is an exception in the message handler, I send it to Application Insights and rethrow (to let the handler fail and process the message again). But I have no evidence of that.

I think I've checked everything, and I cannot tell why those messages disappeared. In your experience, what should that be?

Is that normal that Azure waits for 5 minutes after shutting down? Or is that a signal that I'm not honoring the cancellation token?

Thanks

2
azureazureservicebusazure-servicebus-queuescancellation-token
What's the MaxDeliveryCount on that queue and have you checked the Transfer DLQ? – Sean Feldman
MaxDeliveryCount is 10. What's the Transfer DLQ? – Marconline
Have a post about it: weblogs.asp.net/sfeldman/… – Sean Feldman
No, no message neither in the Transfer queue nor in the Transfer DLQ. – Marconline
Hmmm, then I'm not sure what's going on. Usually messages do not disappear w/o any traces unless that connection string & entity is used by some other code... is that an option? – Sean Feldman

2 Answers

1
votes

As we know, the receiver can read a message from a Service Bus queue in two different ways: ReceiveAndDelete and Peek-lock. The ReceiveAndDelete mode, if the receiver crashes before it finishes processing the message, the message will be lost. The Peek-lock mode, if the receiver crashes before it finishes processing the message and do not complete the message, the message will not be lost/disappear.

As others said, not seeing your code, it is hard to find the issue. If possible, please modify your code to set OnMessageOptions.AutoComplete property to false, and explicitly call Complete() after the receiver finishes processing the message and creating record on your DB, and then check if it could solve the issue.

0
votes

I've just experienced similar symptoms with on a function app processing from a Service Bus Topic Subscription where messages seemed to be disappearing.

I was instantiating ServiceBusClient via the AzureClientServiceCollectionExtensions.AddAzureClients in my function app startup. Inside the class where I was creating a ServiceBusReceiver in ReceiveAndDelete mode and calling the ReceiveMessagesAsync method.

The problem seems to be with the ServiceBusReceiver hanging around supposedly dequeuing messages in the background (even after my function app had stopped!). The fix for this was to dispose of the Receiver once I was done with it.