Hundreds of Azure Function Instances to Process Event Hub Events

Question

My case is as follows, processing one event takes from 30s up to 3min. I am creating/(event hub is receiving) around 100k-300k events every day. What I created is an Event Hub with 32 partitions (max available for standard plan), an Consumer plan Azure function.

But the problem is that, the Azure function is scaling up to 32 instances, which is not enough. The workaround that I can think of is to create another Function App and connect it to the same Consumer Group. But that does not sound like a reasonable solution.

Can I increase the number available Function Instances in any other way that is more appropriate?

Answer 1

As already described by others (as well as in this: Kafka how to consume one topic parallel) parallelism on read from one Event Hub (a Kafka topic) is limited to number of partitions on it.

• One Consumer may consume from one or more Partitions.
• One Partition may be consumed by at most one Consumer at a time.

If you assign more then "extra" consumers would just wait till they're assigned a partition if/when some "active" consumer dies.

---

Other options for better throughput are:

(I assume you don't have a choice but to consume from Event Hub to begin with)

• Pay more. Get a Dedicated plan. Limit is 1024 partitions. Though it's debatable from documentation so I've raised a bug to clarify.
• Do something about 3 minutes, it really is slow.
• Consume in batches: If your function's code is more I/O bound then let it consume more than one messages at once.
  • Note that this can be a little tricky, because all params you specify are "suggestions" to Azure Function Runtime (that polls assigned partition(s) on your behalf) and you don't really have a fine grained control over number of messages per batch. Usually your problem would be too small a batch size.
• Introduce another Q (storage Q, blob storage, Service Bus) between the code that consumes from Event Hub (say af_eh_consumer) and code that processes the message (say a new Azure Function af_msg_processor. So af_eh_consumer would just post message to storage Q, which would trigger af_msg_processor, this trigger is not limited by limit of 32. Problems with this:
  • Error handling: You'll have to understand how your new Q's retries work as you'll checkpoint (move the pointer) your Event Hub once af_eh_consumer has posted it to storage Q.
  • Order: Storage Q does not guarantee order of messages, if you care about that then use an ordered Q like Service Bus.
  • More coding, maintenance, ops efforts.

Answer 2

Why wouldn't an instance per partition not be enough? More instances will only compete to get a lock on the same partitions. So it won't increase performance. There should be an instance per partition.

It might be better to invest in faster processing. Three minutes per event is incredible long for such a system. Can't the handling of such events be delegated to another system

If some events take long you might be able to use more than one consumer group to make a distinction between fast and slow to process events and take it for granted that that de slow consumer group has some lag in the processing of the events.

Answer 3

I believe you are on the wrong messaging service. Azure Event Hubs is not a good fit for your scenario. Data throughput is low and your case sounds like you need to avoid duplicate processing as much as possible since it is costly to reprocess the same message multiple times. Therefore, I recommend you to look at Service Bus Queue triggers for Azure Functions below.

https://docs.microsoft.com/en-us/azure/azure-functions/functions-bindings-service-bus-trigger?tabs=csharp

Answer 4

How about increasing the number of worker processes within an instance? https://docs.microsoft.com/en-us/azure/azure-functions/functions-best-practices#use-multiple-worker-processes

Those workers share the same resources within an instance but at least you have more worker threads that can process requests.