kafka Consumer Upgrade to .9

Question

Hi I am working on kafka upgrade to .9 and also upgrading the kafka consumer to new java consumer released with .9. While upgrade, i am using the existing topics, Steps are just stoping the .8 kafka and Starting the .9 kafka pointing to same log.dirs, At consumer side, i am using the same group name and topic name,But the new consumer is consuming the messages again from starting position in the topic. I have committed them in .8. I am adding auto.offset.reset = earliest.

Any idea why it is happening or We need to first consume all the message with .8 consumer and let the lag to become 0. Thanks Sunny

Answer 1

While 0.8 consumers store offsets in Zookeeper, 0.9 consumers store them in a specific Kafka topic. Consequently a 0.9 consumer will not normally see any offsets stored by a 0.8 consumer and as such will start consuming from start. There is a FAQ that may help in migrating the offsets. To quote:

1) Upgrade your brokers and set dual.commit.enabled=false and offsets.storage=zookeeper (Commit offsets to Zookeeper Only).

2) Set dual.commit.enabled=true and offsets.storage=kafka and restart (Commit offsets to Zookeeper and Kafka).

3) Set dual.commit.enabled=false and offsets.storage=kafka and restart (Commit offsets to Kafka only).

(Have not tried this, just quoting.)