Azure Queues - Functions - Message Visibility - Workers?

Question

I have some questions regarding the capabilities regarding Azure Queues, Functions, and Workers. I'm not really sure how this works.

Scenario:

• q-notifications is an queue in an Azure storage account.
• f-process-notification is a function in Azure that is bound to q-notifications. Its job is to get the first message on the queue and process it.

In theory when a message is added to q-notifications, the function f-process-notification should be called.

Questions:

1. Does the triggered function replace the need to have workers? In other words, is f-process-notification called each time a message is placed in the queue.

2. Suppose I place a message on the queue that has a visibility timeout of 5 minutes. Basically I am queueing the message but it shouldn't be acted on until 5 minutes pass. Does the queue trigger f-process-notification immediately when the message is placed on the queue, or will it only trigger f-process-notification when the message becomes visible, i.e. 5 minutes after it is placed on the queue?

Answer 1

In Azure Functions, each Function App instance running your queue triggered function will have its own listener for the target queue. It monitors the queue for new work using an exponential backoff strategy. When new items are added to the queue the listener will pull multiple items off of the queue (batching behavior is configurable) and dispatch then in parallel to your function. If your function is successful, the message is deleted, otherwise it will remain on the queue to be reprocessed. To answer your question - yes we respect any visibility timeout you specify. If a message is added with a 5 minute timeout it will only be processed after that.

Regarding scale out - when N instances of your Function App are running they will all cooperate in processing the queue. Each queue listener will independently pull batches of messages off the queue to process. In effect, the work will be load balanced across the N instances. Exactly what you want :) Azure Functions is implementing all the complexities of the multiple consumer/worker pattern for you behind the scenes.

Answer 2

I typically use a listener logic as opposed to triggers. The consumer(s) are constantly monitoring the queue for messages. If you have multiple consumers, for example 5 instances of the consuming code in different Azure worker roles processing the same bus/queue, the first consumer to get the message wins (they are "competing"). This provides a scaling scenario common in a SOA architecture..

This article describes some of the ways to defer processing.

http://markheath.net/post/defer-processing-azure-service-bus-message

good luck!