Azure Function CosmosDBTrigger unscaleable

Question

I have a Azure Function with a CosmosDBTrigger working on a consumption play monitored with Application Insights. The collection being monitored has 500,000 inserts in the change feed. The consumption plan scales up the number of instances to 15 within a few minutes but only the first instance is able to process any changes.

I assume this is because of the lease is kept by the first instance. I am essentially paying for 14 instances to do nothing.

I noticed you are supposed to be able to set a property on CosmosDBTrigger called LeaseOptions but I currently get an error whenever I try: "Error CS0655 'LeaseOptions' is not a valid named attribute argument because it is not a valid attribute parameter type"

Is there a way to scale a CosmosDBTrigger Azure Function so it can be processed by 10, 20 or even 200 instances at a time?

Answer 1

Yeah, they do expose those LeaseOptions on the attribute as you noticed, but basic .NET 101 rules prohibit a custom complex type like ChangeFeedHostOptions being stored on an attribute at compile time, hence the compile time error you get. If you were using a WebJob you could configure those settings manually as you start up the job host, but it's just not addressable under the Azure Function model.

That said, when you don't configure it, it gets created with the defaults behind the scenes which are as follows:

CheckpointFrequency      null 
FeedPollDelay            00:00:05 
LeaseAcquireInterval     00:00:13 
LeaseExpirationInterval  00:01:00 
LeasePrefix              null 
LeaseRenewInterval       00:00:17

So, given that, I would expect that the longest you should see a single function instance hold onto the entire partition is 17s since every 13s the client is supposed to check the number of hosts (in this case function instances) petitioning for work and rebalance the partitioning amongst them. The fact that you're not seeing this is... perplexing. 🤔

Disclaimer: I have not personally worked with the CosmosDbChangeTrigger in this capacity; I'm simply looking at the code tagged 3.0.0-beta4 in the GitHub repo to figure out how they connect all this up in the function extension's implementation.

Answer 2

As Drew mentioned, the LeaseOptions are not configurable, but it's something that is coming soon.

As for the scaling, the Trigger will start sharing leases following the Change Feed Processor Library's design among the instances to balance the load. This happens within a couple of seconds that the instances appear and there is a limit on how many instances will be distributed based on the amount of leases. Since each lease represents a Partition Key Range, at maximum, they will be distributed 1 lease per instance. Adding more instances over the amount of Partition Key Ranges will not add more parallel processing.

It could be that the instances are being created / destroyed sooner than the algorithm can share the leases.

Are the 500,000 inserts spread among several partitions or are all in the same partition? Is the Function lagging behind the changes or is it processing them at good speed? Do you have other Functions (either running locally or in Azure) monitoring the same collection using the same lease collection?

As a side note, you are not paying for the amount of instances, but the amount of executions.