Queue messages not being moved to the poison queue

Question

I have a job that imports files into a system. Everytime a file is imported, we create a blob in azure and we send a message with instructions to a queue so that the data is persisted in SQL accordingly. We do this using azure-webjobs and azure-webjobssdk.

We experienced an issue in which after the messages failed more than 7 times, they didn't move to the poision queue as expected. The code is the following:

Program.cs

public class Program
{
    static void Main()
    {
        //Set up DI
        var module = new CustomModule();
        var kernel = new StandardKernel(module);

        //Configure JobHost
        var storageConnectionString = AppSettingsHelper.Get("StorageConnectionString");
        var config = new JobHostConfiguration(storageConnectionString) { JobActivator = new JobActivator(kernel), NameResolver = new QueueNameResolver() };
        config.Queues.MaxDequeueCount = 7;
        config.UseTimers();

        //Pass configuration to JobJost
        var host = new JobHost(config);
        host.RunAndBlock();
    }
}

Functions.cs

public class Functions
{
    private readonly IMessageProcessor _fileImportQueueProcessor;

    public Functions(IMessageProcessor fileImportQueueProcessor)
    {
        _fileImportQueueProcessor = fileImportQueueProcessor;
    }

    public async void FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
    {
        await _fileImportQueueProcessor.ProcessAsync(item);
    }

}

_fileImportQueueProcessor.ProcessAsync(item) threw an exception and the message's dequeue count was properly increased and re-processed. However, it was never moved to the poison-queue. I attached a screenshot of the queues with the dequeue counts at over 50.

After multiple failures the webjob was stuck in a Pending Restart state and I was unable to either stop or start and I ended up deleting it completely. After running the webjob locally, I saw messages being processed (I assumed that the one with a dequeue count of over 7 should've been moved to the poison queue). Any ideas on why this is happening and what can be done to have the desired behavior.

Thanks,

Update Vivien's solution below worked.
Matthew has kind enough to do a PR that will address this. You can check out the PR here.

Vivien Chevallier Vivien Chevallier · Accepted Answer · 2017-01-26T16:34:39

Fred,

The FileImportQueue method being an async void is the source of your problem.

Update it to return a Task:

public class Functions
{
    private readonly IMessageProcessor _fileImportQueueProcessor;

    public Functions(IMessageProcessor fileImportQueueProcessor)
    {
        _fileImportQueueProcessor = fileImportQueueProcessor;
    }

    public async Task FileImportQueue([QueueTrigger("%fileImportQueueKey%")] string item)
    {
        await _fileImportQueueProcessor.ProcessAsync(item);
    }
}

The reason for the dequeue count to be over 50 is because when _fileImportQueueProcessor.ProcessAsync(item) threw an exception it will crash the whole process. Meaning the WebJobs SDK can't execute the next task that will move the message to the poison queue.

When the message is available again in the queue the SDK will process it again and so on.

Queue messages not being moved to the poison queue

1 Answers