How to manage offsets in KafkaItemReader used in spring batch job in case of any exception occurs in mid of reading messages

Question

I am working on a Kafka based spring boot application for the first time. My requirement was to create an output file with all the records using spring batch. I created a spring batch job where integrated with a customized class which extends KafkaItemReader. I don't want to commit the offsets for now as i might need to go back read some records from already consumed offsets. My consumer config has these properties;

enable.auto.commit: false auto-offset-reset: latest group.id:

There are two scenarios-> 1. A happy path, where i can read all the messages from kafka topic and transform them and then write them to an output file using above configuration. 2. I am getting an exception while reading thru' the messages, and i am not sure how to manage the offsets in such cases. Even if i go back to rest the offset, how to make sure it is the correct offset for messages. I dont persist the payload of message record anywhere except it goes to the spring batch output file.

Answer 1

You need to use a persistent Job Repository for that and configure the KafkaItemReader to save its state. The state consists in the offset of each partition assigned to the reader and will be saved at chunk boundaries (aka at each transaction).

In a restart scenario, the reader will be initialized with the last offset for each partition from the execution context and resume where it left off.