Kafka fails to keep track of last-commited offset

0
votes

Is there any known issue with kakfa-broker in managing the offsets? Bcz, problem which we are facing is when we try to restart of kafka-consumer(i.e, app restart) sometimes all the offset are reset to 0. Completely clueless on why are consumers not able to start from the last commited offset.

We are eventually facing this issue in prod wherein the whole q events are replayed again :

  • spring-boot version -- 2.2.6 release
  • spring-kafka - 2.3.7 release
  • kafka-client -2.3.1
  • apache-kafka - kafka_2.12-2.3.1

We have 10 topics with 50 partitions for each topic which belongs to same group, we increase topic-partition and consumer count at run-time based on load.

  • auto-commit = false
  • sync commit each offset after processing
  • max-poll-records is set to 1

After all this config it runs as expected in local setup, after deployed to prod we see such issues nut not at every restart. Is there any config that i'm missing. Completely Clueless!!!!!

2
apache-kafkakafka-consumer-apispring-kafka

2 Answers

0
votes

You need to make sure that:

1) You are using the same Consumer Group ID

2) auto.offset.reset is set to latest

spring.kafka.consumer.group-id=your-consumer-group-id
spring.kafka.consumer.auto-offset-reset=latest

In case you are still seeing this issue, try to enable auto-commit

spring.kafka.consumer.enable-auto-commit=true

and if the issue goes away then it means that your manual commits are not working as expected.

0
votes

Do not enable auto commit per the suggestion in another answer; the listener container will more reliably commit the offsets and, as you say, you don't have the problem all the time.

Is it possible that you receive no records for a week?

Or, is it possible that your broker has a shorter offsets.retention.minutes property?

In 2.0, it was changed from a 1 day default to 1 week. If the offsets have been removed because they expired and you restart the consumer, you'll get the behavior you observe.