Azure WebJob Process Storage Queue Messages Serially

Question

We have a need to process messages from an Azure Storage Queue serially, one at a time.

By default, the JobHostConfiguration.Queues.BatchSize is set to 16

https://github.com/Azure/azure-webjobs-sdk/blob/0581fb1610d56a597523fcea67733944efba9541/src/Microsoft.Azure.WebJobs.Host/JobHostQueuesConfiguration.cs#L15

And those 16 messages the WebJob pulls off the queue will be processed in parallel. Instead, we need the WebJob to process the messages serially, one at a time.

How can this be achieved?

We are currently forced to set the BatchSize to 1, which is not ideal because it introduces a lot of spin up/down time around reading from the message queue.

Clarification: We do not need to enforce first in first out, or guaranteed order processing, just one at a time processing.

I don't believe azure storage queues guarantee first in first out. Have you tried using a service bus queue instead. This page mentions some of the differences: azure.microsoft.com/en-us/documentation/articles/… — Jason Haley
Is the question asking for ordered processing, or just one-at-a-time processing? — David Makogon
@DavidMakogon just one-at-a-time processing, no need for FIFO — Aaron Hoffman

vivek kv vivek kv · Accepted Answer · 2016-08-08T06:43:55

JobHostConfiguration config = new JobHostConfiguration();
config.Queues.BatchSize = 1;
config.Queues.MaxDequeueCount = 1;
config.Queues.MaxPollingInterval = TimeSpan.FromSeconds(5);

Set your BatchSize and MaxDequeueCount to 1, Set MaxPollingInterval to 5 seconds.

Azure WebJob Process Storage Queue Messages Serially

1 Answers