How to prevent Azure webjob processing same message multiple times concurrently

Question

I have an Azure WebJob project, that I am running locally on my dev machine. It is listening to an Azure Service Bus message queue. Nothing going on like Topics, just the most basic message queue.

It is receiving/processing the same message multiple times, launching twice immediately when the message is received, then intermittently whilst the message is being processed.

Questions:

How come I am receiving the same message multiple times instantly? It seems that it's re-fetching before a PeekLock is applied?
How come the message is being re-received even though it is still being processed? Can I set the PeekLock duration, or somehow lock the message to only be processed once
How can I ensure that each message on the queue is only processed once?
I want to be able to process multiple messages at once, just not the same message multiple times, so setting MaxConcurrentCalls to 1 does not seem to be my answer, or am I misunderstanding that property?

I am using an async function, simple injector and a custom JobActivator, so instead of a static void method, my Function signature is:

public async Task ProcessQueueMessage([ServiceBusTrigger("AnyQueue")] MediaEncoderQueueItem message, TextWriter log) {...}

Inside the job, it is moving some files around on a blob service, and calling (and waiting for) a media encoder from media services. So, whilst the web job itself is not doing a lot of processing, it takes quite a long time (15 minutes, for some files).

The app is launching, and when I post a message to the queue, it responds. However, it is receiving the message multiple times as soon as the message is received:

Executing: 'Functions.ProcessQueueMessage' - Reason: 'New ServiceBus message detected on 'MyQueue'.'
Executing: 'Functions.ProcessQueueMessage' - Reason: 'New ServiceBus message detected on 'MyQueue'.'

Additionally, whilst the task is running (and I see output from the Media Service functionality), it will get another "copy" from the queue.

finally after the task has completed, it's still intermittently processing the same message.

What's the DeliveryCount and LockDuration specified on the queue? — Sean Feldman
I need to look into it. Weird thing about the LockDuration is that it's seemingly picking up the message twice almost instantly, as if it's not locking it at all. I wonder if it's actually an issue with the way I have set up the asynchronous handler? — AndrewP
Sounds a bit off. The broker will never give the same message more than to one competing consumer if LockDuration has not expired. I suspect something else is happening. Any chance you could share a repro on GitHub or BitBucket? — Sean Feldman

Sean Feldman Sean Feldman · Accepted Answer · 2016-08-04T13:18:24

I suspect what's happening is the following: Maximum DurationLock can be 5 minutes. If processing of the message is done under 5 minutes, message is marked as completed and removed from the broker. Otherwise, message will re-appear if processing takes longer than 5 minutes (we lost the lock on the message) and will be consumed again. You could verify that by looking at the DeliveryCount of your message.

To resolve that, you could renew message lock just before it's about to expire using BrokeredMessage.RenewLockAsync().

How to prevent Azure webjob processing same message multiple times concurrently

1 Answers