kafka custom consumer to read incoming records

Question

I have created a kafka consumer using the new API (http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html)

Currently the consumer is reading reading the messages from the topic starting from the smallest offset. I want to override this to read from the latest offset. Any pointers on how this can be done?

Answer 1

Once you start reading messages for a given consumer group, you'll need to explicitly force the consumer to the end of the partitions it's following with the seekToEnd() method. For consumer groups that haven't yet read any messages, you can use the auto.offset.reset=largest property to tell the consumer where to read its first message.

Answer 2

Try to set auto.offset.reset = largest, although this should be the default setting.