Retrieve a message by knowing its partition and offset in Kafka

Question

I'm working on Kafka 0.9. I'm wondering if there is any approach to retrieve a message, which has been processed, from its topic by knowing the partition and offset. For example, the consumer is currently consuming the message at partition 1 and offset 10. And I want to get the message at the same partition and offset 5.

One way that I can think of is to reset the offset to 5 and consume one single message. But the poll() method can only return a batch of messages. So I have to take the first message and disregard the others. After processing the message, the offset is reset back.

I think this will work. But still want to know if there is any other elegant way of doing it.

Answer 1

Kafka is designed to read long stripes of data off of the disk without moving the disk heads around -- in other words, it is optimized to use linear reads. It seems inefficient to disregard a whole chunk of data you had to read off of disk (and possibly serve over the network) but it is actually a lot more inefficient to make the disk head jump around a lot. Check out Kafka's design philosophy, and about it's use of disks, here.

In other words, your approach probably works. But you are thinking more like the way someone uses a relational database, not a messaging system.

Answer 2

You should be able to use the "seek" method to read the message from the offset you require.

Take a look at the "Controlling the Consumer's Position" https://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html