Kafka Consumer behaviour when trying to commit after a rebalance has happened due to some other consumer failure?

Question

What would happen to commitSync() or it's variants when a rebalance has happened due to failure of not the current consumer but because of failure of some other consumer in the same group.

Say, consumer-1 (c1) was assigned partitions 1 & 2 (p1 and p2) out of say a total of 10 partitions. c1 has done a poll() and fetched 200 records from p1 (offset 400 to 500) and p2(offset 1300 to 1400), then processed them and is about to commit. But between c1's poll() and commit some other consumer has failed and a re-balance has happened. And C1 got assigned to partitions p4 and p6. Now will c1 be still able to commit the offsets to p1 and p2 (which he was assigned to earlier) or will it lead to an exception like CommitFailedException ?

Maybe this answers your question: stackoverflow.com/questions/60924842/… — Michael Heil
@Mike, from the above - "This can happen when a group rebalance completes before the commit could be successfully applied." - this addresses my issue — joven

abhaybhatia abhaybhatia · Accepted Answer · 2021-04-25T02:25:11

From java doc

public void commitSync()
Commit offsets returned on the last poll() for all the subscribed list of topics and partitions.
This is a synchronous commits and will block until either the commit succeeds or an unrecoverable error is encountered (in which case it is thrown to the caller).

For commitSync, if a rebalance occurs before the commitSync is called, the commit will fail with an exception.
Hence in your case, the messages will be read again by the new consumer that gets assigned to partitions P1 and P2

One way to get more control over rebalances is to implement ConsumerRebalanceListener which has methods for
a. onPartitionsRevoked
b. onPartitionsAssigned

Kafka Consumer behaviour when trying to commit after a rebalance has happened due to some other consumer failure?

1 Answers