Scheduled Script to calculate total consumer lag Kafka

Question

I want to write script which calculates total lag per consumer/partition. I know general command.

bin/kafka-consumer-groups -bootstrap-server :9092, :9092,:9092 --describe --group

What is best way to parse the data and print it to log? Along with topic name and consumer lag I will add timestamp every time this command is run. so I can send data to Elastic search and make a metric of total lag versus time per consumer.

We dont want to use tools kafka manager and burrow Kafka version 1.1.0

Thanks,

I think this can be accomplished using simple bash commands piped to the above mentioned command. — Abhishek Kumar
Have you figured this out? Can you get it for older kafka versions? — s g

RigaCrypto RigaCrypto · Accepted Answer · 2019-10-31T08:54:25

My own personal approach is the show the consumer groups and then select only the LAG part with a grep command that pipes into an awk command that sums the LAG values.

So just try to add this at the end:

| grep -E "LAG|*" |  awk '{SUM += $5} END { print SUM}'

You can play with it a little if awk $5 doesn't match the LAG column. Hope this helps and works :D

Scheduled Script to calculate total consumer lag Kafka

1 Answers