How to check consumer offsets when the offset store is Kafka?

10
votes

From 0.8.1.1 release, Kafka provides the provision for storage of offsets in Kafka, instead of Zookeeper (see this). I'm not able to figure out how to check the details of offsets consumed, as the current tools only provide consumer offset count checks for zookeeper only.(I'm referring to this)

If there are any tools available to check consumer offset, please let me know.

3
apache-kafka
The OffsetChecker works well with offsets stored in Kafka. - codejitsu
@leshkin Are you talking about the kafka.tools.ConsumerOffsetChecker that comes bundled in the /bin? - Abhiram
yes, see my answer for details. - codejitsu

3 Answers

8
votes

I'am using kafka 0.8.2 with offsets stored in kafka. This tools works good for me:

./kafka-run-class.sh kafka.tools.ConsumerOffsetChecker 
        --topic your-topic 
        --group your-consumer-group
        --zookeeper localhost:2181

You get all informations you need: topic size, consumer lag, owner.

4
votes

The following straight command gives the enough details:

kafka-consumer-groups.sh --bootstrap-server localhost:9092 --describe --group my-second-application

You will get the details like this

TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
first_topic     0          4               4               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first_topic     1          3               3               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first_topic     2          3               3               0               consumer-1-7cb31cf3-1621-4635-8f95-6ae85215b31b /10.200.237.53  consumer-1
first-topic     0          4               4               0               -                                               -               -
2
votes

I'm using Kafka 2.1 and I use kafka-consumer-groups command which gives useful details like current offset, log-end offset, lag, etc. The simplest command syntax is

kafka-consumer-groups.sh            \
--bootstrap-server localhost:29092  \
--describe --group <consumer group name>

And the sample output looks like this

TOPIC       PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID            HOST      CLIENT-ID
your.topic  1          17721650        17721673        23   consumer-159-beb9050b  /1.2.3.4  consumer-159
your.topic  3          17718700        17718719        19   consumer-159-beb9050b  /1.2.3.4  consumer-159
your.topic  0          17721700        17721717        17   consumer-159-beb9050b  /1.2.3.4  consumer-159

HTH