Kafka consumer lag changes randomly?

Question

So I have a kafka consumer on version 0.9.0.1 and I often run the following command to check on it:

kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --zookeeper <my-zookeeper> --group my-group

Recently I've noticed that my lag randomly changes and I have no idea why, for example, if I ran the command twice I'd get:

Group   Topic    Pid Offset     LogSize    Lag   Owner
<group> <topic>  0   148802905  148865604  62699 <my-consumer>
...

But on the second run, immediately after I'd get:

Group   Topic    Pid Offset     LogSize    Lag   Owner
<group> <topic>  0   148865241  148865715  474   <my-consumer>
...

I also have a similar issue with another consumer group, but instead of the lag changing numbers randomly, it switches between a number and the text unknown

Where should I look? seems like a problem with kafka but may be a consumer issue as well, any ideas welcome. Thanks in advance

Do you mean the offset and log size reported are not correct? Could the reason be just spikes on production or consumption rate? — vahid
you might be hitting different data centers or servers by any chance? Look at your CNAME? — Fakrudeen

sandeshch sandeshch · Accepted Answer · 2018-06-14T04:54:07

if Offset+lag = logSize then its fine. lag might increase or decrease based on your consumers rate of consumption and acking it back. You should check the consumer why it becomes slow intermediately.Often happens when the consumer is performing some operation and then acking the message back. We use kafka with storm and face the similar issue when some downstream (e.g. persistence) is taking time.

Kafka consumer lag changes randomly?

1 Answers