kafka offset management auto vs manual

Question

I'm working on an application of spring boot which uses Kafka stream, in my application, I want to manage Kafka offset and commit the offset in case of the successful message processing only. This is important, to be certain I won't lose messages even if Kafka restarted or the zookeeper is down. my current situation is when my Kafka is down and up my consumer starts from the beginning and consumes all the previous messages.

also, I need to know what is the difference between managing the Kafka offset automatic using autoCommitOffset and manging it manually using HBase or zookeeper or checkpoints?

also, what are the benefits of managing it manually if there is an automatic config we can use?

Answer 1

You have no guarantee of durability with auto commit

Older Kafka clients did use Zookeeper for offset storage, but now it is all in the broker to minimize dependencies. Kafka Streams API has no way to integrate offset storage outside of Kafka itself, and so you must use the Consumer API to lookup and seek/commit offsets to external storage, if you choose to do so, however, you can still then end up with less than optimal message processing.

my current situation is when my Kafka is down and up my consumer starts from the beginning and consumes all the previous messages

Sounds like you set auto.offset.reset=earliest and you never commit any offsets at all...

The auto commit setting does a periodic commit, not "automatic after reading any message".

If you want to guarantee delivery, then you need to set at least acks=1 in the producer and actually do a commitSync in the consumer