Azure service bus Azure Function trigger and get all messages

5
votes

I have a queue that a lot of high frequency data is put in. I have an azure function that will trigger when new messages are written to it and write the message to a Azure sql database. My problem is that there are too many messages to write so what I want to do is kind of "accumulate" messages and then write them all at once to the database. I don't see any kind of functionality in Azure function service bus integration to get all(or x) messages and handle them.

Is something like this at all possible?

Any help is appreciated.

4
c#azureazure-functionsazureservicebus
You're looking for batching, and it isn't currently supported. There is an open github issue requesting the feature: github.com/Azure/azure-webjobs-sdk/issues/1024 – McGuireV10
Also, you probably won't ever see a "get all" variation -- because of the way SB partitions queues to multiple queue instances, Azure itself never accurately knows the "real" number of messages queued for delivery (for example, while it's grabbing a count from partition #3, somebody could write to the already-counted partition #1). – McGuireV10
Also, just in case you run across it, SB queue Functions rather confusingly refers to multiple instances as "batching"... see this answer. – McGuireV10

4 Answers

1
votes

My way of solving this so far is to avoid service bus trigger because it involves that the function executes one instance per message. I do a time trigger every 10 seconds and I use the method Receive from the subscription client.

Indeed in this case you would need to manually bring the Azure Service Bus library, create the needed classes and call the Receive method which you can tell how many items you want. Do follow best practice that ASB classes (TopicClient and SubscriptionClient) should be set as singletons because they are expensive objects.

I also have to accumulate messages to sort them and deduplicate them so this is how I solved it. However, it would be very cool to have that kind of functionality out of the box from Azure Functions extension.

-2
votes

I'm just wondering why you need that batch insert. for every single message that triggers your function a new instance of you function is being created by Azure; so performance won't be an issue here. If you still wants to do batching, you may store incoming messages temporarily in a proxy Azure database/data store and query that proxy in specified periods and run your batch insert command. Please keep in mind that you must delete queried records after successful batch insert.

-2
votes

Azure Functions runtime retrieves and processes queue messages in batches. And the default batchSize is 16 and the maximum batchSize is 32..This can be configured for 'queue' triggers by mentioning batchSize in host.json file of the Function App. Refer to know more on host.json

Configuration settings for 'queue' triggers

"queues": {
  "maxPollingInterval": 2000,
  "visibilityTimeout" : "00:00:10",
  "batchSize": 16,
  "maxDequeueCount": 5,
  "newBatchThreshold": 8
}