Say if auto-commit interval time is 30 seconds, consumer for some reasons could not process the message and hold it longer than 30 seconds then crash. does the auto-commit offset mechanism commits this offset anyway right before consumer crash?
If my assumption is correct, the message is lost as its offset committed but the message itself has not been processed?
Lets consider your Consumer group name is Test and you have a single consumer in the Consumer Group.
When Auto-Commit is enabled, offsets are committed only during poll() calls and during closing of a consumer.
For example- auto.commit.interval.ms is 5 secs, and every call to poll() takes 7 secs. When making every call to poll(), it will check if the auto commit interval has elapsed, if it has, like in the above example, it will commit the offset.
Offsets are also committed during closing of a consumer.
From the documentation -
"Close the consumer, waiting for up to the default timeout of 30 seconds for any needed cleanup. If auto-commit is enabled, this will commit the current offsets if possible within the default timeout".
You can read more about it here -
https://kafka.apache.org/10/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html
Now, onto your question, if poll() is not called again or consumer is not closed, it won't commit the offset.
If the Consumer receives message N, commits it and then crashes before having fully processed it then by default the Consumer will considered this message processed.
Note that the message is still on the broker, so it can be re-consumed to be processed. But that require some logic in your application to not only restart from last committed position but also check if previous records were processed successfully.
If your application typically takes a long time to process messages, maybe you want to switch to manual commit instead of auto. That way you'll be able to better control when you commit and avoid this issue.
