Durable function based design for long running activity

0
votes

As per business requirement, we need to on broad customer's data of third party AD in to our application database & then need to establish real time sync with third party AD to sync up with their thereafter changes.

To do that, our side APIs and third party side APIs are ready. Since this on-boarding & real time sync activities happens on demand (not 24x7 but whenever there is change on the third party AD resource or whenever we acquire new client) I prefer to do Azure function based solution but Azure function is having max. timeout of 10mins on consumption plan which I don't believe fits to our requirement. At the moment we are not consider Azure function's other hosting plan.

So thought of doing Durable function orientated design where notification about recent changes on third party AD will be received in Azure queue which triggers durable function execution, in turn orchestrator function will query third party for recent of list of changes on user resource & the response may return user changes in 1000s in numbers.

Then orchestator loop through the user change list and invoke brand new activity function for every single user in the list which in turn calls out in house REST API(will accept only one user object not list of users as its parameter) & onboard the user to our application.

By this design we may end up with thousand instances of same activity function running in our azure account at same time. Is this possible?

Any limitation in turn max number instantiation of particular activity function on given point time?

Is every activity function is nothing but regular Azure function?

Are there better way designing instead of spinning up new instance of activity function for every new onboarding user resource?

Long running Activity function possible with Durable extension of Azure function which alone will take care of activity lasting for more than 30 minutes under consumption plan? please suggest.

1
azure-functionsazure-durable-functions

1 Answers

1
votes

Q: By this design we may end up with thousand instances of same activity function running in our azure account at same time. Is this possible?
A: Yes. Your scenario is commonly referred to as the fan-out/fan-in pattern.

Q: Any limitation in turn max number instantiation of particular activity function on given point time?
A: There are a couple of performance targets documented that you can refer to but do note that they will depend on what your activity functions do.

Also, you can try different throttle settings to improve the numbers but beware of the consequences mentioned in the doc.

Q: Is every activity function is nothing but regular Azure function?
A: Yes. Think of it like a queue triggered function

Q: Are there better way designing instead of spinning up new instance of activity function for every new onboarding user resource?
A: The consumption plan should be able to easily handle such scenarios, but note that the fan-in operation runs on a single VM as mentioned in the performance targets doc. If this operation requires processing (like aggregation), you could consider using sub-orchestrations to split it across multiple VMs.

Q: Long running Activity function possible with Durable extension of Azure function which alone will take care of activity lasting for more than 30 minutes under consumption plan?
A: Not possible with the consumption plan. You will have to use the Premium Tier or an App Service Plan.