Azure function running multiple times for the same service bus queue message

0
votes

I have an Azure function (based on the new C# functions instead of the old .csx functions) that is triggered whenever a message comes into an Azure Service Bus Queue. Once the function is triggered, it starts processing the service bus message. It decodes the message, reads a bunch of databases, updates a bunch of others, etc... This can take upwards of 30 minutes at times.

Since, this is not a time sensitive process, 30 minutes or even 60 minutes is not an issue. The problem is that in the meanwhile, Azure Function seems to kick in again and picks up the same message again and again and reprocesses it. This is an issue and cause problems in our business logic.

So, the question is, can we force the Azure function to run in a singleton mode? Or if that's not possible, how do we change the polling interval?

4
c#azuresingletonazureservicebusazure-functions
Was your Function App created under the App Service Plan or Consumption Plan? Under App Service Plan, make sure that AlwaysOn is turned on. Under the Consumption Plan, the Function App is terminated after 5 minutes (configurable to extend to 10 minutes, see "functionTimeout" value at github.com/Azure/azure-webjobs-sdk-script/wiki/host.json), which may explain the behavior you are seeing. - Ling Toh
I have the App Service Plan. My function is not terminating. In fact, the opposite is happening. Multiple functions are processing the message. - Yasir

4 Answers

4
votes

The issue is related to Service Bus setting...

What is happening is that the message is added to the queue, the message is then given to the function and a lock is placed on that message so that no other consumer can see/process that message while you have a lock on it.

If within that lock period you do not tell service bus that you've processed the file, or to extend the lock, the lock is removed from the message and it will become visible to other services that will then process that message, which is what you are seeing.

Fortunately, Azure Functions can automatically renew the lock for you. In the host.json file there is an autoRenewTimeout setting that specifies for how long you want Azure Functions to keep on renewing the lock for.

https://github.com/Azure/azure-webjobs-sdk-script/wiki/host.json 

"serviceBus": {
  // the maximum duration within which the message lock will be renewed automatically.
  "autoRenewTimeout": "00:05:00"
},
4
votes

AutoRenewTimeout is not as great as suggested. It has a downside that you need to be aware of. It's not a guaranteed operation. Being a client side initiated operation, it can and sometimes will fail, leaving you in the same state as you are today.

What you could do to address it is to review your design. If you have a long running process, then you process the message, and hand off processing to something that can run longer than MaxLockDuration. The fact that your function is taking so long, indicates you have a long running process. Messaging is not designed for that.

One of the potential solutions would be to get a message, register processing intent in a storage table. Have another storage table triggered function to kick-off processing that could take X minutes. Mark it as a singleton. By doing so, you'll be processing your messages in parallel, writing "request for long running processing" into storage table, completing Service Bus messaging and therefore not triggering their re-processing. With the long running processing you can decide how to handle failure cases.

Hope that helps.

0
votes

So your question is on how to make the Azure Function Message trigger to process one message at a time, and a message has to be processed once only.

I have been able to achieve the same functionality using the following host.json configuration.

{
  "serviceBus": {
    "maxConcurrentCalls": 1,
    "autoRenewTimeout": "23:59:59",
    "autoComplete": true
  }
}

Note that I set the autoRenewTimeout to 24 hour, which is long enough for a really long running process. Otherwise you can change it to a duration that fits your need.

Many will argue the suitability of using Azure Function for a long running operation. But that is not the question that needs an answer here.

0
votes

I also experience the same issue, what I did is remove the default rule and add a custom rule on subscriber. 

(OriginalTopic='name-of-topic' AND OriginalSubscription='name-of-subcription-sub') OR (NOT Exists([OriginalTopic]))