Is an Azure Function PULL pattern possible using Storage Queue?

0
votes

I currently have an Event Grid-triggered Azure Function which executes whenever blobs are uploaded to a specific storage account. This will not scale as larger and larger blobs are uploaded. There is also the potential for "losing" messages if the Function ack's them but fails for some reason downstream.

A message queue would allow for buffering so the Function doesn't exhaust memory, and allow for storage of messages until a confirmed completion occurs.

I'm looking for an example of a PULL pattern where the Function processes queue messages on its own terms.

Thinking:

  1. When blobs are uploaded to the Storage Account,
  2. Event Grid write messages to a Storage Queue,
  3. Function polls the Storage Queue for unprocessed messages,
  4. Processes messages accordingly
  5. Upon success, dequeue's message
  6. If fail, retry message

Something like this: enter image description here

Is this a valid approach or would I just use a Queue Storage trigger for the Function and eliminate Event Grid from the scenario?

1
pythonazure-functionsazure-storage-queues

1 Answers

1
votes

Event Grid has the concept of Event Handlers of which Azure Functions is one option.

For your use case, you could simply forward the events to a storage queue or service bus queue for processing by a function app that reads from the queues instead.

Both queue providers have their own retry mechanism that Azure Functions already runs by.