Azure Queue delayed message

Question

I has some strange behaviour on production deployment for azure queue messages: Some of the messages in queues appears with big delay - minutes, and sometimes 10 minutes. Befere you ask about setting delayTimeout when we put message to queue - we do not set delayTimeout for that message, so message should appear almost immedeatly after it was placed in queue. At that moments we do not have a big load. So my instances has no work load, and able to process message fast, but they just don't appear.

Our service process millions of messages per month, we able to identify that 10-50 messages processed with very big delay, by that we fail SLA in front of our customers.

Does anyone have any idea what can be reason?

How to overcome?

Did anyone faced similar issues?

Answer 1

Some general ideas for troubleshooting:

1. Are you certain that the message was queued up for processing - ie the queue.addmessage operation returned successfully and then you are waiting 10 minutes - meaning you can rule out any client side retry policies etc as being the cause of the problem.

2. Is there any chance that the time calculation could be subject to some kind of clock skew problems. eg - if one of the worker roles pulling messages has its close out of sync with the other worker roles you could see this.

3. Is it possible that in the situations where the message is appearing to be delayed that a worker role responsible for pulling the messages is actually failing or crashing. If the client calls GetMessage but does not respond with an appropriate acknowledgement within the time specified by the invisibilityTimeout setting then the message will become visible again as the Queue Service assumes the client did not process the message. You could tell if this was a contributing factor by looking at the dequeue count on these messages that are taking longer. More information can be found here: http://msdn.microsoft.com/en-us/library/dd179474.aspx.

4. Is it possible that the number of workers you have pulling items from the queue is insufficient at certain times of the day and the delays are simply caused by the queue being populated faster than you can pull messages from the queue.

5. Have you enabled logging for queues and then looked to see if you can find the specific operations (look at e2elatency and serverlatency). http://blogs.msdn.com/b/windowsazurestorage/archive/tags/analytics+2d00+logging+_2600_amp_3b00_+metrics/. You should also enable client logging and try to determine if the client is having connectivity problems and the retry logic is possibly kicking in.

And finally if none of these appear to help can you please send me the server logs (and ideally the client side logs as well) along with your account information (no passwords) to JAHOGG at Microsoft dot com.

Jason

Answer 2

Azure Service bus has a property in the BrokeredMessage class called ScheduledEnqueueTimeUtc, it allows you to set a time for when the message is added to the queue (effectively creating a delay).

Are you sure that in your code your not setting this property, and this might be the cause for the delay?

You can find more info on this at this url: https://www.amido.com/azure-service-bus-how-to-delay-a-message-being-sent-to-the-queue/

Answer 3

If you are using WebJobs to process messages from the queue, it can be due to WebJobs configuration.

From an MSDN forum post by pranav rastogi:

Starting with 0.4.0-beta, the (WebJobs) SDK implements a random exponential back-off algorithm. As a result of this if there are no messages on the queue, the SDK will back off and start polling less frequently.

The following setting allows you to configure this behavior.

MaxPollingInterval for when a queue remains empty, the longest period of time to wait before checking for a message to. Default is 10min.

static void Main()
{       
    JobHostConfiguration config = new JobHostConfiguration();       
    config.Queues.MaxPollingInterval = TimeSpan.FromMinutes(1);        
    JobHost host = new JobHost(config);
    host.RunAndBlock(); 
}