Kaka auto.offset.reset query

0
votes

My project uses Kafka 0.10.2 version. Iam setting enable.auto.commit=false and auto.offset.reset=latest in the consumer. If consumer is restarted after maintenance, the consumer is reading again from first offset instead of waiting for latest offset messages. Any reasons why is this happening? Have i understood the configurations wrongly?

My requirement is the consumer should not auto commit and should read only the new messages put into the topic when it is active.

2
apache-kafkakafka-consumer-api
Have you checked your logs to validate if the configurations were set exactly as you wished? If nothing could be found there, and answer below doesn't solve it, you should post your code of your consumer. – mike

2 Answers

0
votes

Just because you aren't auto committing doesn't guarnatee there are no manual commits.

Regardless, auto.offset.reset=latest will never send the consumer group to the beginning of the topic. Sounds like whatever Kafka tool / library you are using is calling a consumer.seekToBeginning call on its own.

0
votes

For Understanding purpose , The Consumer property auto.offset.reset determines what to do if there is no valid offset in Kafka for the Consumer’s Consumer Group Based on the below scenarios :

– When a particular Consumer Group starts the first time

– If the Consumer offset is less than the smallest offset

– If the Consumer offset is greater than the last offset

▪ The value can be one of:

– earliest: Automatically reset the offset to the earliest available

– latest: Automatically reset to the latest offset available

– none: Throw an exception if no previous offset can be found for the ConsumerGroup

▪ The default is latest