options for handling message processing in azure service bus queue

Question

I am working on a command processing application which uses azure service bus queue. Commands are issued from a website and posted to the queue and the queue messages are processed by a worker role. Processing involves fetching data from db and other sources based on the queue message values and sending it to different topics. The flow is ,

1. Receive message
2. process the message
3. Mark message as complete / Abandon message On processing exception.

The challenge I face here is the processing time. Sometimes it exceeds the maximum message lock time period (5 minutes -configured) and hence the message is unlocked and it re-appears for the worker role to pick up (consider multiple instances of the worker role). So this causes same message to be processed again.

What are the options I have to handle such a scenario.?

I have thought about ,

1. Receive message - add to a local variable - mark message complete. In case of exception send the message again to the queue or to a separate queue (let us say failed message queue). A second queue also means another worker role to process it.

2. In the processing there is a foreach loop that runs. So I thought of using a Parallel.Foreach instead . but not sure how much of time gain it will give and also read some posts on issues when using Parallel in azure.

Suggestions,fixes welcome.

Answer 1

Aravind, you can absolutely use SB queue in this scenario. With the latest SDK you can renew the lock on your message for as long as your are continuing to process it. Details are at: http://msdn.microsoft.com/en-us/library/microsoft.servicebus.messaging.brokeredmessage.renewlock.aspx

This is similar to the Azure storage queue functionality of updating the visibility timeout: http://msdn.microsoft.com/en-us/library/windowsazure/microsoft.windowsazure.storage.queue.cloudqueue.updatemessage.aspx

Answer 2

You may want to consider using an Azure Queue, the maxium lease time for an Azure Queue message is 7 days, as opposed to the Azure Service Bus Queue lease time of 5 minutes.

This msdn article describes the differences between the two Azure queue types.

If the standard Azure Queue doesn't contain all the features you need you might consider using both types of Queue.

Answer 3

You can fire off a Task with a heartbeat operation that keeps renewing the lock for you while you're processing it. This is exactly what I do. I described my approach at Creating a Task with a heartbeat