Scale CosmosDB binding for Azure Functions per logical partition

1
votes

I would like my Azure function to scale per logical partition instead of per physical partition. I've tested the Azure Function binding and it does scale out when I have multiple physical partitions (in my test I needed to increase our RU's from 2000 to 20000). But I don't need that much RU since I'm using it as an event store. I'm not querying the data, just processing each message through my Azure function. So I'm wondering if there is a way to let Azure Functions scale out per partition. I see that in the new v3 lib there is a ChangeFeedOptions.PartitionKey property but that class is internal and I'm not sure it does what I want.

I basically want to have as many Azure Functions running as there are new messages grouped per logical partition. What would be the best way to achieve that?

1
azureazure-functionsazure-cosmosdb
im very curious to know your use case. what operation exactly is the function performing? i've built a very configurable throttler for azure functions, something that definitely is not available OOB – Alex Gordon
We use the Azure function to process the events from our event store to create a projection (an optimized for reading data storage; see 'event sourcing' for more info). Or do you mean what the operation is performing on a functional level? – Zenuka
Indeed I was asking on a functional level however I'd love to collaborate more on the event driven solution. What are you using for an event store? So basically you're using cosmosdb as the trigger for different operations in your services? How do you choose when to write to cosmosdb vs writing to table storage? – Alex Gordon
We've got a stream of employee data (could be about 50.000 events) coming in for a specific customer. It might take a while to process that so we don't want other customers to wait for those events to be processed. We hoped to use Cosmos as our event store because of the change feed but since it doesn't exactly do what we need (on our tiny scale), we're probably going with table storage and some other form of queue to get projections updated (probably service bus topics since we've got that running already). – Zenuka
what exactly do you need to do that change feed does not support? – Alex Gordon

1 Answers

1
votes

As of today this is not possible. It's not up to the user of the CF SDK to do the lease management. The CF SDK will do that for us and there is nothing we can do to change it.

The only way to theoretically actually have one lease per logical partition is to have a logical partition big enough to occupy the whole of a physical partition. This however means that you are about to hit 10GB of data in a single partition which would be the main concern you would have at this point.

I wouldn't worry about the scaling though. The CF will spawn as many leases as it needed to scale seamlessly and this scaling depends solely on the volume of data in the database and the amount of RUs allocated.