Offset commit on Spring Kafka while setBatchListener(true) with enable.auto.commit true

2
votes

Could someone please help to understand when the consumer thread will commit the offset with spring kafka batch listener and auto commit set to true? Will the consumer thread commits the offsets after processing all messages in the batch as default? I know if autoCommit is false, the commit will be based on the AckModes, but would like to know when autoCommit is true

  • Spring Kafka - batchListener -true
  • enable.auto.commit - true
  • MAX_POLL_RECORDS_CONFIG - default (500)
  • MAX_POLL_INTERVAL_MS_CONFIG - default (5 min)
1
kafka-consumer-apispring-kafka

1 Answers

3
votes

With enable.auto.commit=true, the container has no responsibility at all for committing offsets - it is entirely up to the algorithm in the kafka-clients library.

From the kafka documentations:

If true the consumer's offset will be periodically committed in the background.

Also see auto.commit.interval.ms

The frequency in milliseconds that the consumer offsets are auto-committed to Kafka if enable.auto.commit is set to true.

It is set to 5 seconds by default so if you process a batch in less than that time, the offsets may, or may not, be committed when the batch is complete. The client checks the time since last commit at the start of the next poll.

I prefer to set it to false so that you have a well-defined behavior - the container will commit the offsets when the batch listener exits normally.