Where are consumer groups list stored in recent Kafka version?

Question

Apparently, in earlier kafka version, the list of consumers for a certain consumer group was stored in zookeeper. Where is this information stored for latest kafka release?

Mickael Maison Mickael Maison · Accepted Answer · 2019-12-21T11:37:12

Since Kafka 0.10, the list of Consumer Groups are stored in the __consumer_offsets topic.

That topic contains both the committed offsets and the groups metadata (group.id, members, generation, leader, ...). Groups are stored using GroupMetadataMessage messages (Offsets use OffsetsMessage).

You can dump the groups metadata using the GroupMetadataMessageFormatter. For example:

./bin/kafka-console-consumer.sh \
  --formatter "kafka.coordinator.group.GroupMetadataManager\$GroupMetadataMessageFormatter" \
  --bootstrap-server localhost:9092 \
  --topic __consumer_offsets

Note that this is a compacted topic, so to get the list of groups, you would need to "materialize" all entries. This is what brokers do when you use the listConsumerGroups() API

Where are consumer groups list stored in recent Kafka version?

2 Answers