Azure ServiceBus queue not retrying messages when TaskCanceledException is thrown

5
votes

I have an Azure WebJob function that listens to messages on an Azure ServiceBus queue. Usually when I encounter an exception in my code, the message is abandoned as per the Azure WebJobs SDK documentation:

The SDK receives a message in PeekLock mode and calls Complete on the message if the function finishes successfully, or calls Abandon if the function fails. If the function runs longer than the PeekLock timeout, the lock is automatically renewed.

According to the Azure ServiceBus documentation this should mean that the message becomes available again, and will be retried:

If the application is unable to process the message for some reason, it can call the AbandonAsync method on the received message (instead of CompleteAsync). This method enables Service Bus to unlock the message and make it available to be received again, either by the same consumer or by another competing consumer. Secondly, there is a timeout associated with the lock and if the application fails to process the message before the lock timeout expires (for example, if the application crashes), then Service Bus unlocks the message and makes it available to be received again (essentially performing an AbandonAsync operation by default).

The behavior described above is what usually happens, but I have found an exception to this rule. If my code throws a TaskCanceledException specifically, the message is not abandoned as it should:

public void ProcessQueueMessage([ServiceBusTrigger("queue")] BrokeredMessage message, TextWriter log)
{
     throw new TaskCanceledException();
}

When running this function through a web job, I see the error message printed out clear as day, but the message is consumed without any retries and without entering the dead-letter queue. If I replace the TaskCanceledException above with InvalidOperationException, the message is abondened and retried as it should (I have verified this against an actual ServiceBus queue).

I have not been able to find any explanation for this behavior. Currently I am wrapping the TaskCanceledException in another exception to work around the issue.

The question

Is what I am experiencing a bug in the Azure WebJobs SDK? Is TaskCanceledException special in this regard, or do other types of exceptions have similar behavior?

I am using the following NuGet packages:

  • Microsoft.Azure.WebJobs 2.3.0
  • Microsoft.Azure.WebJobs.ServiceBus 2.3.0
1
c#azureazureservicebusazure-webjobsazure-servicebus-queues

1 Answers

0
votes

Functions are supposed to abandon the message if execution was not successful. If you're saying the message was not abandoned and retried even though it should be (assuming MaxDeliveryCount was set to larger than 1 and the receive mode was PeekLock), then it's likely to be the issue with the Functions and not Azure Service Bus. You could verify that by running a console application and performing the same, checking wherever the message is completed and removed from the queue or still on the queue and available for consumption.

Also, looks like you're using the older version of WebJobs (and Azure Service Bus). When performing verification, you'd need to use the older Azure Service Bus client (WindowsAzure.ServiceBus) and not the new one (Microsoft.Azure.ServiceBus).