Kafka consumer lag through JMX

2
votes

I'm trying to monitor the lag of a consumer group in Kafka 0.10.

Our consumers are keeping track of their offsets in Kafka rather than ZooKeper. This means I can get the figures using the following:

bin/kafka-consumer-groups.sh --bootstrap-server <broker> --describe --group <group-name>

This works fine. However, my broker already makes use of the Prometheus JMX exporter for collecting a number of stats. I've connected JConsole to the brokers but can't see the same data being reported in JMX as reported by kafka-consumer-groups.sh.

Is there anyway to get this information from Kafka with JMX without needing any additional tools?

1
apache-kafkajmx
Possible duplicate of Kafka Consumer - JMX Properties – Alex Ott
"Kafka: The Definitive Guide" recommends to use Burrow: engineering.linkedin.com/apache-kafka/… – Alex Ott
It appears both that book and Burrow share an author. Was there any specific reason why Burrow was suggested? We already have a alerting/monitoring tools in place so I don't need that aspect, just the consumer lag information. – GuessBurger

1 Answers

0
votes

You could retrieve the atrributes {topic}-{partition}.records-lag of metric kafka.consumer:type=consumer-fetch-manager-metrics,client-id={client-id} for all partitions. That should be equivalent to the output of consumer-groups.sh