After kafka crashed, the offsets are lost

3
votes

Our kafka system crashed because no disk space was available. The consumers are Spring boot application which are using the Kafka Streams API. Now every consumer application shows the following error:

java.io.FileNotFoundException: /tmp/kafka-streams/908a79bc-92e7-4f9c-a63a-5030cf4d3555/streams.device-identification-parser/0_48/.checkpoint.tmp (No such file or directory)

This exception occurred exactly after the kafka server was restarted. If we restart the application, the service starts at offset 0 to replay all message from the consumed topic. This is fatal.

We checked the logs: On each partition there is no information (example line, we have 60 partitions):

TOPIC                               PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                                                                                                            HOST            CLIENT-ID
device-identification.received      0          -               0               -               streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer-ca723291-3258-46c0-8b23-233bdc1e1191 /XX.X.X.X       streams.device-identification-parser-3e81ef05-b9e3-495c-85b3-7e2912f00578-StreamThread-1-consumer

Can anyone help me, to restore the offset information, that the service starts from the last offset?

1
javaapache-kafkakafka-consumer-apiapache-kafka-streamsspring-kafka
Would resetting the offsets manually for all the customers work for you? Something like described here: stackoverflow.com/questions/29791268/… - t6nn
Hi, in your link, I see, that it is possible to set the offset by datetime. I will try it with my team tomorrow in the morning CET. This comment is helful, thanks! - Norbert Koch
It works fine! I was able to restore the offset time based. Thanks! - Norbert Koch

1 Answers

1
votes

If this is a Kafka Streams app you might be successful in resetting it, that can be done using Confluent's Application reset tool: https://docs.confluent.io/current/streams/developer-guide/app-reset-tool.html