How to get the last offset time from zookeeper? when reading messages from kafka using storm spout. The context: Kafka gets messages continuously, consumer reads for a while and then shuts down due to any reason and then the consumer reads only the latest message but does not read from the last offset read
1 Answers
2
votes
consumer reads for a while and then shuts down due to any reason
Not sure what exactly are you referring to as the consumer is supposed to run infinitely unless it is stopped explicitly.
Now assuming you are using the storm's KafkaSpout implementation, there is a config called forceStartOffsetTime which is used to force the spout to rewind to a previous offset. The way to use it as follows
spoutConfig.forceStartOffsetTime(-2);
As seen in the doc page
It will choose the latest offset written around that timestamp to start consuming. You can force the spout to always start from the latest offset by passing in -1, and you can force it to start from the earliest offset by passing in -2.
so setting it to -2 will always force it to read from the start what is the configuration you are using, would be great if you could post some code
