How to stop an Azure WebJobs queue message from being deleted from an Azure Queue?

1
votes

I'm using Azure WebJobs to poll a queue and then process the message.

Part of the message processing includes a hit to 3rd party HTTP endpoint. (e.g. a Weather api or some Stock market api).

Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??

If I continue .. then I assume the message will be deleted by the WebJobs SDK.

How can I:

1) Say to the SDK - please don't delete this message (so it will be retried automatically at the next queue poll and when the message is visible again).
2) Set the invisibility time value, when the SDK pops a message off the queue for processing.

Thanks!

1
azureazure-webjobs
Based on the content posted here azure.microsoft.com/en-in/documentation/articles/…, I believe the message is not deleted if your function throws an error. Can you try by logging the exception and then rethrowing that exception? – Gaurav Mantri
Assuming you dont continue/handle error but allow it to fail - and if your request fails more than 5 times -the message is pushed into another queue what is called a posion-{{your_queue_name}} - you can write a listener listening to this queue to process the failed message. Please clarify what you mean by invisibility value? You mean a lock. – Jaya
@JS_GodBlessAll Reading the documentation, 5 is the default value for the number of retries but is configurable. If we set this value to a really high number, then the message won't move to poison queue (see Manual poison message handling section). Am I correct? – Gaurav Mantri
@GauravMantri : Am not sure about configuring retries cause when I used SDK version (3 months back) it did not let me. However I could configure and did configure 1) Polling interval 2) and the no of messages allowed to be dequeued - this is what I did. May be the newer version let you configure the failures too – Jaya
Also @GauravMantri and @ Pure.Krome - I believe (vaguely remember) adding another dll to achieve the same - that might be for CRON jobs - not sure though, also these values are capped at a max value - somewhere hidden in the docs . – Jaya

1 Answers

8
votes

Now, if the hit to the api fails (network error, 500 error, whatever) I try/catch this in my code, log whatever and then ... what??

The Webjobs SDK behaves like this: If your method throws an uncaught exception, the message is returned to the Queue with its dequeueCount property +1. Else, if all is well, the message is considered successfully processed and is deleted from the Queue - i.e. queue.DeleteMessage(retrievedMessage);

So don't gracefully catch the HTTP 500, throw an exception so the SDK gets the hint.

If I continue .. then I assume the message will be deleted by the WebJobs SDK.

From https://github.com/Azure/azure-content/blob/master/articles/app-service-web/websites-dotnet-webjobs-sdk-get-started.md#contosoadswebjob---functionscs---generatethumbnail-method:

If the method fails before completing, the queue message is not deleted; after a 10-minute lease expires, the message is released to be picked up again and processed. This sequence won't be repeated indefinitely if a message always causes an exception. After 5 unsuccessful attempts to process a message, the message is moved to a queue named {queuename}-poison. The maximum number of attempts is configurable.

If you really dislike the hardcoded 10-minute visibility timeout (the time the message stays hidden from consumers), you can change it. See this answer by @mathewc:

From https://stackoverflow.com/a/34093943/4148708:

In the latest v1.1.0 release, you can now control the visibility timeout by registering your own custom QueueProcessor instances via JobHostConfiguration.Queues.QueueProcessorFactory. This allows you to control advanced message processing behavior globally or per queue/function.

https://github.com/Azure/azure-webjobs-sdk-samples/blob/master/BasicSamples/MiscOperations/CustomQueueProcessorFactory.cs#L63

protected override async Task ReleaseMessageAsync(CloudQueueMessage message, FunctionResult result, TimeSpan visibilityTimeout, CancellationToken cancellationToken)
{
      // demonstrates how visibility timeout for failed messages can be customized
      // the logic here could implement exponential backoff, etc.
      visibilityTimeout = TimeSpan.FromSeconds(message.DequeueCount);

      await base.ReleaseMessageAsync(message, result, visibilityTimeout, cancellationToken);
}