I'm currently trying to test the kafka retention log using a new environment with docker-kafka.
I used config/server.properties and set the following for log retention:
log.retention.ms=2000
log.retention.check.interval.ms=2000
Creating a topic and adding messages to it, I would test the size of the log by going to the location of the output logs. In my case it was at /tmp/kafka-logs/<topic-name>. Then just ls -l to see size in bytes.
Adding more messages will increase size in bytes of the log file. If there is a better way to check the logs, please let me know.
Running:
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic <topic-name>
I would get no config output. Log file is not deleted after 2000 ms.
On Kafka docs:
The number of milliseconds to keep a log file before deleting it (in milliseconds), If not set, the value in log.retention.minutes is used
However, setting a topic-level configuration with:
$ bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic <topic-name> --config retention.ms=2000
And checking configs on the title:
$ bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic <topic-name>
I see that the retention is set for the topic. Also, checking the log size again, it might delete the logs after 2000 ms, but it does clear within a short time.
How can I set default configurations for all topics created? Specifically related to log retention time?
Also, an additional add-on question, are there config files for each individual topic created? I'm mainly asking because I know how to set topic-level configurations via cli, but was curious to see if these topic-level configs were saved somewhere.
