Prevent multiple instances of an Azure web job processing the same Queue message

Question

I noticed that multiple instances of my Web job are receiving the same message and end up acting on it. This is not the desired behavior. I would like multiple messages to be processed concurrently, however, I do not want the same message being processed by multiple instances of the web job.

My web job is of the continuous running type.
I use a QueueTrigger to receive the message and invoke the function
My function runs for several hours.

I have looked into the JobHostConfiguration.BatchSize and MaxDequeue properties and I am not sure on these. I simply want a single instance processing a message and that it could take several hours to complete.

This is what I see in the web job logs indicating the message is received twice.

[01/24/2017 16:17:30 > 7e0338: INFO] Executing: 'Functions.RunExperiment' - Reason: 'New queue message detected on 'runexperiment'.'

[01/24/2017 16:17:30 > 7e0338: INFO] Executing: 'Functions.RunExperiment' - Reason: 'New queue message detected on 'runexperiment'.'

Not sure if this is the best solution, but I got the idempotency issue resolved by inspecting the dequeueCount to know whether to run or exit out. I was hoping that the message queue would default, or allow configuration, to prevent a fanout of the message. — Arif
Did you see any exceptions ? When exceptions are thrown, messages are abandonned. — Thomas
There were no exceptions thrown. Two instances could be seen running in webjob logs and they would both complete and duplicate the work. — Arif
Oh, if it takes a while to process a message, you should renew the lock. you can use ` OnMessageOptions.AutoRenewTimeout`. see stackoverflow.com/a/33785389/4167200 — Thomas

Tom Sun - MSFT Tom Sun - MSFT · Accepted Answer · 2017-01-25T09:18:13

According to the official document, if we use Azure queue storage in the WebJob on the multiple instance, we no need to write code to prevent multiple instances to processing the same queue message.

The WebJobs SDK queue trigger automatically prevents a function from processing a queue message multiple times; functions do not have to be written to be idempote.

I deployed a WebJob on the 2 instances WebApp, It also works correctly(not execute twice with same queue message). It could run on the 2 instances and there is no duplicate executed message.

So it is very odd that the queue message is executed twice, please have a try to debug it whether there are 2 queue messages that have the same content are triggered.

The following is my debug code. I wrote the message that with the executed time and instance id info into another queue.

public static void ProcessQueueMessage([QueueTrigger("queue")] string message, [Queue("logqueue")] out string newMessage, TextWriter log)
    {
        string instance = Environment.GetEnvironmentVariable("WEBSITE_INSTANCE_ID");
        string newMsg = $"WEBSITE_INSTANCE_ID：{instance}, timestamp：{DateTime.Now}, Message：{message}";
        log.WriteLine(newMsg);
        Console.WriteLine(newMsg);
        newMessage = newMsg;
    }
}

Prevent multiple instances of an Azure web job processing the same Queue message

2 Answers