Kafka high level consumer

Question

I am trying to use the High level consumer for batch reading the messages in the Kafka topic. During this batch read, my thread has to stop at some point.

Either, once all the messages in the topic are exhausted. or Get the max offset at the point when the messages are about to be read and stop till that max offset is reached.

I tried to use the code at high-level-consumer but the iterator methods on the KafkaStream seems to be a blocking call and waits till another messages comes in.

So 3 questions,

1. How do i know that there are no more messages to be read from the topic?

2. If i have an answer to the above question, how do i stop it from listening to the topic anymore?

3. Is there a way to find the maximum offset when the batch read starts (i think simple consumer can do this) and make the high level consumer stop at that point?

Answer 1

You have the option to decide that when no new message has arrived for a specified amount of time, then you consider all messages to have been read. This can be set using the consumer property consumer.timeout.ms. After this specified value has passed without any new messages arriving, the ConsumerIterator will throw a timeout exception and you could handle that in the consumer and exit.