Spring Kafka consuming old messages which are already consumed by the consumer

Question

i have a Spring boot application and using Spring Kafka. we have create a consumer which is consuming messages from 4 topics. these topics doesnt have any partition. the issue i am facing here a rendom behavior that out of three topics, in any one topic offset stop and my consumer keep on consuming same messages from that topic again and again until we need to manually move the offset to latest.below is the configuration YAML configuration i have :

spring:
  kafka:
   consumer:
      bootstrap-servers:  ${KAFKA_BOOTSTRAP_SERVERS}
      group-id: group_id
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer
kafka:
  consumer:
    allTopicList: user.topic,student.topic,class.topic,teachers.topic**

as it is a Spring boot application, default offset is set to latest. what i am doing wrong here, please help me to understand.

Gary Russell Gary Russell · Accepted Answer · 2020-04-27T13:53:25

What version are you using?

You should set

...consumer:
     enable-auto-commit: false

The listener container will more reliably commit the offsets.

You should also consider

     ack-mode: RECORD

and the container will commit the offset for each successfully processed record (default is BATCH).

Spring Kafka consuming old messages which are already consumed by the consumer

1 Answers