Why can't I see consumers in Kafka 0.9.0.1?

Question

I've got a Kafka app that can create multi-partition topics, populate them, and read from them with a consumer. That seems to work.

I'm trying to get visibility into Kafka so I've tried a few tools like kafka-manager and trifecta. These tools will tell me about my topics, partitions, etc., but don't seem to see my consumers to get the offsets for them.

If I look at my topics in kafka I see my topic plus a __consumer_offsets, which I think means Kafka (not Zookeeper) is storing my consumers.

Because this is a problem consistent across tools I'm wondering if I don't have something properly set in kafka, or are the tools not yet fully supporting 0.9.x?

Answer 1

In Kafka 0.9, you can try utility kafka-consumer-groups.sh which replaces the old command line utility ConsumerOffsetChecker. That should provide offset information of consumer.

In Kafka 0.9, it has default topic named __consumer_offsets to store offset instead of zookeeper. In older version, you can use offsets.storage consumer property to control whether to use zookeeper or kafka to store consumer offset and zookeeper was the default value. kafka is the preferred method since zookeeper is not optimized for concurrent write.