What may cause huge load in Kafka `__consumer_offsets` topic?

3
votes

I have simple observation in my Kafka cluster (Kafka 0.11.0.0).

According to jmx information the __consumer_offsets topic constantly is loaded with 10 times more messages than sum of all messages in all other topics. I had also conected console consumer to this topic and I can measure similar values.

  • What could be the reason?
  • How can I check what is Kafka broker doing and generate such a load on its own?
2
performanceapache-kafkaloadinternalconsumer
if you read __cosumer_offsets topic, you can know what groupId do load. That should help in research – Natalia
Thanks for the hint. But unfortunately messages in this topic are binary and I do not see anything :-(. – Seweryn Habdank-Wojewódzki

2 Answers

8
votes

to read __consumer_offsets topic:

bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server brokers --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --new-consumer --consumer.config consumer.conf

for kafka 11 use formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

where consumer.conf has one line

exclude.internal.topics=false

1
votes

For version 2.2 of kafka, this is a demo:

kafka_2.12-2.2.1/bin # 

./kafka-console-consumer.sh --bootstrap-server  your-broker   --topic __consumer_offsets --formatter  "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

Output will be:

[MyGroup-7cc5f948df-f9tqz,__MY_TOPIC,2]::OffsetAndMetadata(offset=30, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1568121107017, expireTimestamp=None)