Kafka Stream and Consumer Group Weird Behavior

Question

I have two high-level questions broken up into more individual questions, both of the high-level questions deal with a consumer group an Apache Kafka Streams API is creating and using.

First of all, is the output from the kafka-consumer-group.sh script. I am getting weird output that doesn't really tell me where a particular consumer is at though they seem to be connected to a particular group/topic/partition:

TOPIC    PARTITION    CURRENT-OFFSET    LOG-END-OFFSET    LAG
STANDARD_DATA                  9          11              11              0          myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-1-consumer-4fd9dc15-d8a7-4598-85a9-3761ae6a747b/1.1.1.1                 myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-1-consumer
STANDARD_DATA                  0          4               11              7          myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-5-consumer-28e1c7bf-860d-44d6-bf58-5e0ff875587c/1.1.1.1                 myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-5-consumer
STANDARD_DATA                  4          -               10              -          myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-4-consumer-a3023af6-eafb-4633-85f1-048c20c4dfb3/1.1.1.1                 myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-4-consumer
STANDARD_DATA                  5          -               10              -          myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-3-consumer-a81f1399-1fc4-4579-b24f-fa8fee01fabf/1.1.1.1                 myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-3-consumer
STANDARD_DATA                  3          -               12              -          myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-2-consumer-6a83bfcc-2c6e-4e9d-a819-029ac8c6ae17/1.1.1.1                 myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-2-consumer
STANDARD_DATA                  8          12              12              0          myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-4-consumer-6d46bed3-70c4-4c7f-8e53-f9591192bc3f/1.1.1.1                 myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-4-consumer
STANDARD_DATA                  7          -               11              -          myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-3-consumer-5313315b-ded9-4fe7-ac9d-d8d5b20dd5b9/1.1.1.1                 myConsumer-13b61e5a-6289-45db-844b-3ef8c5a26782-StreamThread-3-consumer
STANDARD_DATA                  2          10              10              0          myConsumer-b9402faf-4b37-479f-82be-a17eaa180c62-StreamThread-1-consumer-c08a648f-548e-47a8-8bc5-7b6fa3bc1fb5/1.1.1.1                  myConsumer-b9402faf-4b37-479f-82be-a17eaa180c62-StreamThread-1-consumer
STANDARD_DATA                  1          2               10              8          myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-2-consumer-08d99679-d430-4e9f-a3b9-11e558ca34a4/1.1.1.1                 myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-2-consumer
STANDARD_DATA                  6          -               12              -          myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-5-consumer-666040f8-d4d0-49e9-9db6-c6efee49ebe1/1.1.1.1                 myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-5-consumer

Why is it that some CURRENT-OFFSETS (3rd column) and LAG (4th column) are showing up as '-' when I can query the API of Kafka directly to distinguish that they are in fact caught up?

(queried through golang API)

4                      myConsumer-7fc71848-465b-4817-93b3-42b9ba290dcd-StreamThread-4-consumer-a3023af6-eafb-4633-85f1-048c20c4dfb3    OFFSET: 10        LOG-END: 10                LAG: 0

Also, why on would that offset not being showing up as represented in the logs (aka, it should be caught up) in general?

My second high level question is that of streams. We have a stream process working that is, at random times (mostly during a restart), resetting to the earliest offset available in particular topics. Throughout the code there is no 'reset', and the OFFSET_RESET isn't touched. I also can confirm that we are not using the 'exactly-once', so I am not sure where exactly these offset resets are coming into play.

Once again, its basically:

stream process is churning through the data, something ~happens~ and then our offsets are back to ground 0, processing again. This can be going on for days to weeks before it decides to reset too, so the committing of offsets IS happening.

Matthias J. Sax Matthias J. Sax · Accepted Answer · 2017-10-02T20:37:35

About the output of kafka-consumer-groups.sh: A - in CURRENT-OFFSET indicates that there is no committed offset for this partition. This implies, that the lag can also not be computed (thus, you get a - there, too).

If I read your statement correctly, if you query the offsets with golang, it shows that partition 4 is at offset 10, in contrast to what kafka-consumer-groups.sh showns -- not sure why this is the case...

About the reseted offsets: You might need to increase broker config offsets.retention.minutes -- default is 24h (cf. https://docs.confluent.io/current/streams/faq.html#why-is-my-application-re-processing-data-from-the-beginning).

Also note that Streams API uses default reset policy "earliest" (in contrast to Consumer API that use "latest" as default). You can change the reset policy in Streams API via StreamsConfig: https://docs.confluent.io/current/streams/developer-guide.html#non-streams-configuration-parameters

Kafka Stream and Consumer Group Weird Behavior

1 Answers