Active consumer not discovered from zookeeper

0
votes

I am new to apache kafka and I have set up a kafka-broker with zookeeper on top of my self-deployed kubernetes cluster. (2 seperate pods)

There is no noticeable error logs and everything works fine including:

  • creating topics
  • producing messages through bin/kafka-console-producer.sh
  • subscribing(consuming) messages through bin/kafka-console-consumer.sh
  • Listing topics in zookeeper CLI ( ls /brokers/ids )

Except:

when I try listing active consumers in zookeeper CLI or view active consumers in the monitoring tool, I get empty results as well.(while I have a consumer session running describe above)

Integration between zookeeper and kafka broker seems to work find since when I try:

[zk: localhost:2181(CONNECTED) 1] ls /brokers/topics
[topic1, topic2] 
[zk: localhost:2181(CONNECTED) 2] ls /consumers
[]

Am I missing something??

1
apache-kafkaapache-zookeeper

1 Answers

0
votes

Consumer data are not stored in Zookeeper anymore (since 0.9). Consumer group data are now stored directly in Kafka.

To check the state of Consumer Groups, you should use the kafka-consumer-groups.sh tool. For example:

./bin/kafka-consumer-groups.sh --bootstrap-server [HOST:PORT] --group [GROUP]

Note that only Consumers that are part of a group that commit offsets are visible. By default the kafka-console-consumer.sh does not. You need to specify --group [GROUP] to enable it.