How to get the relationship of consumer group and topics

2
votes

I'm using kafka 0.10 kafkaConsumer API to get consumers and topic set under consumer subscribed.

Now I can get topics and partitionIds successfully by topiclist method.

To get consumer group data, I'm not found method under KafkaConsumer, but I can get group list from zookeeper by "zookeeper.getChildren(ZkUtils.ConsumersPath(), false);".

My question is how to get the relationship of group and topics. I know one group can subscribe one or more topics, but I don't know how to get the data by KafkaConsumer API or from zookeeper.

I had tried kafkaConsumer.subscription() but return is an empty list.

what I reference: https://kafka.apache.org/0101/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

1
apache-kafkakafka-consumer-api
What you exactly want? All consumer groups and for each all topics it did subscribe to? Btw: KafkaConsumer#subscription() return the subscription of this single consumer -- if you did not subscribe to any topic yet (not sure if you did or not), it will return an empty list of course. - Matthias J. Sax
Did you ever invoke KafkaConsumer.subscribe() or assign() method before calling subscription()? - amethystic
Thank you Sax. Sorry for description not so clear. I'm not do subscribe. There is an exist kafka cluster. I just want to get its consume group and each group's related topics. @MatthiasJ.Sax - Hanqing
Thank you amethystic. No I didn't invoke subscribe() and assign(). Now I know if never invoke subscribe or assign will get nothing from subscription. Ignore subscribe... I just want to get the exist consumer group and its related topics by API.@amethystic - Hanqing
I had got the topics under consumer group from zookeeper :zookeeper.getChildren(ZkUtils.ConsumersPath()+"/[groupId]/owners", false); Thanks all guys! - Hanqing

1 Answers

3
votes

There is no API in KafkaConsumer to get consumer group information. KafkaConsumer is designed to consume messages from topics and be part of a consumer group. But nothing else.

On command line, you can use bin/kafka-consumer-groups.sh to get consumer group information. Internally, it leverages kafka.admin.ConsumerGroupCommand -- you could try to integrate this in your Java code.

You could also got one lever deeper and use non-public APIs (just have a look into the code of kafka.admin.ConsumerGroupCommand to see how it works internally -- however, this is of course fragile if you want to upgrade later on. Non-public APIs can change for a newer version of course.