Is there any way to get the eldest available offset for a kafka topic

Question

I have a requirement where I am storing information of the offset till which system has read the data. So next time when the system starts reading data again from kafka I need to read data in between the older offset that we have in the system to the newest offset. But the older offset might be invalid due to kafka retention policy. So if we specify older offset in kafka consumer what will be the behavior? Also is there any way we could get the oldest offset value for a particular topic/offset so that we start reading from it?

Answer 1

Simply perform seekToBeginning during start up.

Implement ConsumerSeekAware or, preferably, extend AbstractConsumerSeekAware.

See here.

Just call seekToBeginning in onPartitionsAssigned.

Answer 2

This depends on the way you configure your consumer. Specifically, the auto.offset.rest parameter. If you set it to earliest your consumer will start consuming from the earliest available offset (if the offset you try to consume from is not valid). This way you don't need to find the oldest offset value, since the consumer behaves like I described. You can find more details here.