Records on Kafka topic are not getting consumed after consumer restart

Question

I faced very weird issue with Kafka consumer. My settings are kind of below.

<prop key="enable.auto.commit">true</prop>
<prop key="auto.commit.interval.ms">10</prop>
<prop key="auto.offset.reset">latest</prop>

There are 2 consumer in my group polling to single topic. I am not doing any manual offset management at consumer end.
Now, when my consumer was down and I had few new messages published on my topic, generally I get those messages consumed when I restarts my consumer.
But for one time I observed that after restarting my consumer, I was not able to consume those new messages(published when consumers were down and not polling) from topic. When I published few more messages, it started reading from new message offset and I lost previous message which were published when my consumer was down.
Please let me know possible cause behind this behavior.

auto.offset.reset isn't true/false. It is latest/earliest/none.... — Justin Pihony
No, that is the topic which is maintained by Kafka itself. It should show your commits. — Justin Pihony
Unfortunately I dont have access to this topic - __consumer_offsets . I will kepp you posted. — Tukaram Bhosale

Kiran Balakrishnan Kiran Balakrishnan · Accepted Answer · 2018-04-10T17:05:05

Consumers by default gets the messages that are published to kafka topic after the connection has been established.Try the producer and kafka console consumer without the --from-beginning parameter

Records on Kafka topic are not getting consumed after consumer restart

2 Answers