How to limit calls to Blob Storage from EventProcessorHost?

Question

I have two Azure Eventhubs, each with 32 partitions. Around 700 events pass through the eventhubs every second, first through one, then through the other. One EventHub has 3 concurrent readers/EventProcessorHost, and the other only has one.

My issue is, that the C# EventProcessorHost seems to make a pretty wild amount of calls to the blob storage, for things like lease management and checkpointing. We're talking on average, 800 calls every second.

We register the EventProcessorHost like this

            _eventProcessorHost = new EventProcessorHost(
                _eventHubOptions.CurrentValue.EventHubName,
                _eventHubOptions.CurrentValue.ConsumerGroupName ?? PartitionReceiver.DefaultConsumerGroupName,
                _eventHubOptions.CurrentValue.EventHubConnectionString,
                _eventHubOptions.CurrentValue.StorageConnectionString,
                _eventHubOptions.CurrentValue.StorageName);

            // Reduce lease renewal throughput for storage accounts to save price
            _eventProcessorHost.PartitionManagerOptions = new PartitionManagerOptions()
            {
                LeaseDuration = TimeSpan.FromSeconds(60),
                RenewInterval = TimeSpan.FromSeconds(50)
            };

            var eventProcessorOptions = new EventProcessorOptions
            {
                InitialOffsetProvider = partitionId =>
                    EventPosition.FromEnd(),
                EnableReceiverRuntimeMetric = true,
                MaxBatchSize = 400,
                PrefetchCount = 400,
            };

            await _eventProcessorHost.RegisterEventProcessorFactoryAsync(_eventProcessorFactory, eventProcessorOptions);

We're looking to limit the amount of calls made to storage. As you can see above, we've also extended the lease duration, from the default of 30 seconds, to the maximum of 60.

We've implemented it, so that checkpointing happens every 2000 messages, which means that, on average, checkpointing happens no more than every 15 seconds per partition.

Is this a normal amount of storage calls to coordinate the EventProcessorHosts? Is there any other dials or knobs I can turn, e.g. adjusting partitions to limit the amount of storage calls?

How many hosts are there running in total? Please note that each host will add storage I/O overhead. — Serkant Karaca
@SerkantKaraca 3 hosts on one EventHub, and one host on the other. I can confirm scaling down to 6 partitions on each eventhub with the same throughput/code drops the requests from 500/sec to 1.5/sec — Gustav Wengel
Drop at storage I/O is expected as you scale the partitions down. Given the workload volume you are processing, I don't see a reason to scale out to 32 partitions. Please note that each partition can support up to 1MB/sec or 1K events/sec. — Serkant Karaca
We expect to run at 32 partitions when going full-scale, and as we didn't expect the storage I/O to scale enough to cost a significant amount of money, we didn't see any reason not to keep 32 partitions open off the bat. — Gustav Wengel

Serkant Karaca Serkant Karaca · Accepted Answer · 2020-01-14T18:31:55

Given that you have already optimized processor host and partition manager options, there are only couple things I can recommend:

Reduce number of partitions. Single partition can process up to 1MB/sec or 1K events/sec. Unless of course consumers are doing heavy compute during handling the events.
Reduce number of hosts if hosts are running idle. This will help to reduce Storage I/O overhead originated from each host.

How to limit calls to Blob Storage from EventProcessorHost?

1 Answers