Storm cannot fetch message from Kafka with the message: Got fetch request with offset out of range

1
votes

I'm newbie in Storm world. In my topology, I am consuming data from Kafka and I use SpoutConfig.

With some tests, I got the following warning messages:

2015-10-01 23:31:51.753 s.k.KafkaUtils [WARN] Got fetch request with offset out of range: [85970] 2015-10-01 23:31:51.755 s.k.PartitionManager [WARN] Using new offset: 0

My SpoutConfig sets ignoreZkOffsets to false so I expect that when a new topology is submitted that it consumes from latest offset which is stored in Zookeeper.

However, when I kill my old topology and submit a new topology, the warning message shows up and the new topology can't consume from the offset. Could you give me a good idea if I forget something?

1
apache-kafkaapache-storm
can you post the code you used to generate your SpoutConfig and your KafkaSpout? – SQL.injection
@SQL.injection Thank you for your comments. Finally I found out what I've missed. I will answer my question below. Please refer to it. :) – obanadingyo

1 Answers

1
votes

Finally, I found out what I missed. Since my topology consumes from two different Kafka sources, I made two KafkaSpout with different SpoutConifg instances each of which is allocated to each KafkaSpout.

But when I instantiate each SpoutConfig instance, I passed same id to each instance. It result in that there is only single znode with the id even if I expect to consume from two different sources.

Therefore, two PartitionManager tries to use the same single znode in which there is an offset info. If one of PartitionManager try to consume new tuple with the offset which is the one of the other side, the warning message shows.