How could Kafka 0.8.2.1 with offsets.storage=kafka still require ZooKeeper?

0
votes

We are setting up a new Kafka project using 0.8.2.1 and want to write consumer offsets exclusively to Kafka. So we've set offsets.storage=kafka and dual.commit.enabled=false in our consumer config. However when we create our consumer connector, it still wants to connect to ZooKeeper:

kafka.consumer.Consumer.createJavaConsumerConnector(config);
// fails with:
// Caused by: java.lang.IllegalArgumentException: requirement failed: 
//      Missing required property 'zookeeper.connect'

I thought perhaps we just needed to specify zookeeper.connect and it would be ignored so I specified an invalid hostname, but that still fails because it does try to connect. We really don't want our consumers to have to connect to ZooKeeper if we can avoid it. So what gives?

1
apache-kafka

1 Answers

6
votes

Seems that ZooKeeper is still a requirement in 0.8.2, but they hope to fix this in 0.8.3. From Gwen Shapira on the mailing list:

The existing consumer uses Zookeeper both to commit offsets and to assign partitions to different consumers and threads in the same consumer group. While offsets can be committed to Kafka in 0.8.2 releases, which greatly reduces the load on Zookeeper, the consumer still requires Zookeeper to manage group membership and partition ownership. The new consumer (which we hope to have ready for 0.8.3 release) will completely remove the Zookeeper dependency, managing both offsets and partition ownership within Kafka itself.