Why can it take so long to delete Kafka topics?

Question

On a 3 node cluster, I created few topics with thousands of messages. I have noticed that it takes long time to delete a topic. It took me more than 14 mins to delete 500 topics.

1. Are there any best practices for topic deletion?
2. Is there any document that explains why it takes so much time to delete a topic ?
3. When I create a topic, Kafka will create a folder under log.dirs. I had 10000 topics; I ran a command to delete all of them. Kafka has deleted all 10000 files from log.dirs but Kafka-topics.sh shows topics that does not exist on the file system with "- marked for deletion".

Answer 1

1. I don't think there are any best practices for deleting a topic in Kafka. As far delete.topic.enable=true is defined in server.properties you can simply delete the topic using

bin/kafka-topics.sh --delete --zookeeper localhost:2181 --topic myTopic

2. If your topics were large enough (and you might possibly had a high replication factor as well), then that's something normal. Essentially, the messages of topics are stored in log files. If your topics are extremely large, then it could take some time in order to get rid of all of these files. I think the proper metric here is the size of the topics you attempted to delete and not the number of topics (You can have 500 topics each of which has 1 message as opposed to 500 topics with e.g. 1TB of messages each).

Answer 2

Kafka topic deletion is not guaranteed to be instant. When you 'delete' a topic, you are actually marking it for deletion. When the TopicDeletionManager next runs, it will then start removing any topics that are marked for deletion. This may also take longer if the topic logs are larger.