Azure Queue Behaving Unexpectedly

Question

I have a PHP based application that uses Azure Queues. My application has one cron that runs on my web server every 20 minutes and it adds up to 2000 messages to Azure Queue storage. It used to put those messages in a single queue but now I use two different queues alternatively putting message in each one. There is also a sleeping period of one tenth of a second after every message is added to Queue to avoid throttling.

I have about 40 VMs on Azure that read from this Queue, process each message and then update the database if any of these messages trigger certain event. It happens once in 100 messages.

These VMs have a single PHP script that has a maximum run time of 900 seconds (set_time_limit(900)) and they run as cron jobs every 15 minutes (900 seconds).

The problem is that every half an hour (sometimes every hour) the VMs won't be able to read anything from the Queue for a few minutes. The message will be empty.

This is the code that I use

$listMessagesResult = $queueRestProxy->listMessages($queue_name);
$messages = $listMessagesResult->getQueueMessages();

The result is simply an empty message. I have the following code to trace error

if (empty($messages)) {
        error_log(print_r($messages, TRUE));
    }

It prints and empty array. Exactly this is in my error log

Array\n(\n)\n

Since the code works most of the time, I can't see what is the issue. Here is how I know that it happens regularly http://i.imgur.com/RE9XtVS.png

Y axis is the number of messages. X axis is time. The message count is taken every minute.

The flat curve after every alternate run of the script that adds message shows the problem. If the code had an issue the behavior won't be so erratic. I can't put my finger on what is wrong or where the problem could be. The code is pretty straight forward and it works most of the time other than those 5-10 minutes of non-activity.

I will appreciate any help on this. Thank you.

One thing I would certainly recommend is that you enable Storage Analytics on your queue service and see if anything funky is going on at the storage level. Once storage analytics is enabled, you can see the data in $logs blob container. — Gaurav Mantri

dinwal dinwal · Accepted Answer · 2014-10-12T18:35:42

With Gaurav's help I was able to nail it down. I had set the initial visibility timeout to be 10 minutes and that is what caused the messages to not be visible for a few minutes after they were added to the queue. I confused "setVisibilityTimeoutInSeconds" configuration with the time it takes a message to reappear in the queue if it is not explicitly deleted.

Azure Queue Behaving Unexpectedly

1 Answers