Kafka Consumer Rebalancing : In-Flight Message Processing is Aborted

Question

So when our application Scales-Up / Scales-Down, The Kafka Consumer Group Rebalances.

For example when the application Scales-Down, one of the consumer is killed and the partitions which were earlier assigned to this consumer is distributed across the other consumers in the group, When this process happens i have errors in my application logs saying the processing of the in flight message has been aborted

I know the entire consumer group pauses (i.e) does read any new messages while the consumer group is rebalancing. But what happens to the messages which were read by the consumers before pausing ? Can we gracefully handle the messages which are currently being processed ?

Thanks in advance!

Answer 1

The messages which were read but not committed will be ignored when consumer rebalance occurs.After the consumer rebalance is completed the consumers will resume consuming from the last committed offset , so you won't be loosing any message.