Kafka Auto-Commit doesn't effect for slower transactions

0
votes

Using Spring Boot Kafka - v 2.2.1.RELEASE (which uses kafka-clients.jar v 2.3.1)

I'm using confluent v 5.1.2 for my Kafka Broker.

I've got enabled the autocommit to true and autocommit interval every 3 seconds.

I've got a spring based listener class (@KafkaListener) and each message processing takes about 6-7 seconds as it's a resource intensive operation.

What I noticed here is the offsets are never committed by the listener and the lag permanently remains as "550".

Because of this (I think), the messages are indefinitely processed by the listener.

When I changed the listener to manually commit the offset, it works fine.

Why is the autocommit not working in my case where a long running transaction is involved?

Is there any settings missing or set to too small/large a number? I'd like to know what's the recommended strategy for such long running transactions.

Thanks!

1
apache-kafkakafka-consumer-apispring-kafka

1 Answers

0
votes

Auto commit runs on the consumer thread so auto.commit.interval.ms should be considered a minimum - the actual commit won't happen until the next poll.

However, Spring recommends not using auto commit and letting the container perform the commits (e.g. with AckMode.RECORD); it is more deterministic and reliable.