Azure Storage Queue - processing messages on poison queue

Question

I've been using Azure Storage Queues to post messages too, then write the messages to a db table. However I've noticed that when an error occurs processing messages on the queue, the message is written to a poison queue.

Here is some background to the setup of my app:

Azure Web App -> Writes message to the queue

Azure function -> Queue trigger processes the message and writes the contents to a db

There was an issue with the db schema which caused the INSERTS to fail. Each message was retried 5 times, which I believe is the default for retrying queue messages, and after the 5th attempt the message was placed on the poison queue.

The db schema was subsequently fixed but now I've no way of processing the messages on the poison queue.

My question is can we recover messages written to the poison queue in order to process them and INSERT them into the db, and if so how?

Answer 1

For your particular problem, I would recommend solution mentioned in question part of this post: Azure: How to move messages from poison queue to back to main queue?

Please note that name of poison queue == $"{queueName}-poison"

Answer 2

You can use azure management studio(cerulean) and move the message from poison queue to actual queue. Highly recommended tool to access queues and blobs and do any production related activity also. https://www.cerebrata.com/products/cerulean

I am just user of the tool and no way affiliated, i recommended because it is very powerful, very useful and makes you very productive.

Click on move and message can be moved to the actual uploaded queue

Answer 3

In my current project I've created something what is called: "Support functions" in the FunctionApp. It exposes a special HTTP endpoint with Admin authorization level that can be executed at any time.

Please See the code below, which solves the problem of reprocessing messages from the poison queue:

public static class QueueOperations
{
    [FunctionName("Support_ReprocessPoisonQueueMessages")]
    public static async Task<IActionResult> Support_ReprocessPoisonQueueMessages([HttpTrigger(AuthorizationLevel.Admin, "put", Route = "support/reprocessQueueMessages/{queueName}")]HttpRequest req, ILogger log,
        [Queue("{queueName}")] CloudQueue queue,
        [Queue("{queueName}-poison")] CloudQueue poisonQueue, string queueName)
    {
        log.LogInformation("Support_ReprocessPoisonQueueMessages function processed a request.");

        int.TryParse(req.Query["messageCount"], out var messageCountParameter);
        var messageCount = messageCountParameter == 0 ? 10 : messageCountParameter;

        var processedMessages = 0;
        while (processedMessages < messageCount)
        {
            var message = await poisonQueue.GetMessageAsync();
            if (message == null)
                break;

            var messageId = message.Id;
            var popReceipt = message.PopReceipt;

            await queue.AddMessageAsync(message); // a new Id and PopReceipt is assigned
            await poisonQueue.DeleteMessageAsync(messageId, popReceipt);
            processedMessages++;
        }

        return new OkObjectResult($"Reprocessed {processedMessages} messages from the {poisonQueue.Name} queue.");
    }
}

Alternatively it may be a good idea to create a new message with the additional metadata (as information that the message has already been processed in the past with no success - then it may be send to the dead letter queue).

Answer 4

You have two options

1. Add another function that is triggered by messages added to the poison queue. You can try adding the contents to the db in this function. More details on this approach can be found here. Of course, if this function too fails to process the message you could check the dequeue count and post a notification that needs manual intervention.
2. Add an int 'dequeueCount' parameter to the function processing the queue and after say 5 retries log the failure instead of letting the message go the poison queue. For example you can send an email to notify that manual intervention is required.

Answer 5

Just point your Azure function to the poison queue and the items in that poison queue will be handled. More details here: https://briancaos.wordpress.com/2018/05/03/azure-functions-how-to-retry-messages-in-the-poison-queue/

Answer 6

Azure Storage Explorer(version above 1.15.0) has now added support to move messages from one queue to another. This makes it possible to move all, or a selected set of messages, from the poison queue back to the original queue.

https://github.com/microsoft/AzureStorageExplorer/issues/1064