Kafka - Consumer Offset Reset Automatically

2
votes

While using Apache Kafka with a Java API High Level Consumer, I get the following error sometimes,

WARNpool-1-thread-4 Auto-commit of offsets {my_topic-2=OffsetAndMetadata{offset=53847, metadata=''}} failed for group my_consumer_group: Offset commit failed with a retriable exception. You should retry committing offsets. The underlying error was: This is not the correct coordinator.

Whenever this error occurs, the consumer offsets are reset to a earlier value, and the records which have already been consumed are getting consumed again.

My Kafka version 0.11.0.0. I have 3 Kafka Brokers. The number of partitions for my topic is 3. The number of partitions for the consumer offsets is 50 (default value). The replication factor of Consumer Offsets is 2.

Is this a bug in 0.11.0.0 ? or am I missing any configuration ?

Thanks.

1
javaapache-kafkaapache-zookeeperkafka-consumer-api
could you share some code or configs? - GuangshengZuo
This is not the correct coordinator indicates that the consumer is now talking to a coordinator that's not in charge of the group anymore. There must be more to the log than this like a long pause that has caused the group to rebalance I would assume. Or you have some strange traffic routing in place I'd guess? - dawsaw

1 Answers

0
votes

This was a bug in Kafka version 0.11.0.0 and was later fixed in version 0.11.0.1.

The following ticket tracks the bug,

https://issues.apache.org/jira/browse/KAFKA-5600