Azure Event Hub - consuming messages one by one

1
votes

Is it the desired state that I have e.g. 300 outgoing messages (taken from azure event hub metrics) even though I just handled only one event? I am curious how to consume events only when the previous one was fully handled and checkpointed stored.

Otherwise, if more than one message is consumed by the consumer and the checkpoint is set after the on_event callback is finished (on_event callback takes 10 s in my case), then adding another consumer to the same consumer group and rebalancing partition ownership cause consuming many duplicates.

I am aware of that window where duplications might happen but I would like to have them as few as possible.

FYI I use python sync consumer.

1
pythonazureazure-eventhubconsumer
Do you want to consume them one-by-one or do I not understand it correctly? – Peter Bons
Exactly that is what I meant. – kamroj

1 Answers

2
votes

If you really want to consume messages one by one then Event Hub is not the right solution. You should use Azure Service Bus Queues

Event Hub is not designed for one by one message processing scenario's. Instead it is build for low latency / high throughput. You might be able to come close by having just one partition (and just one consumer in just one consumer group) but you will alway have to risk of processing an already processed event in case the checkpointing fails or something like that.

If you worry about duplication the be aware that most messaging solutions have an at-least-once delivery model of messages (source). So make sure you processing logic is idempotent.