Does KStream autocreate topics in kafka 2.2?

Question

The KStream/TopicNameExtractor javadocs indicate that when using to(TopicNameExctractor extractor) that topics will not be auto-created and must, instead, already exist. I am, however, seeing otherwise with 2.2.0. Are the docs not up-to-date? Can this be relied on with >= 2.2.0?

The javadoc mentioned is here: https://kafka.apache.org/22/javadoc/org/apache/kafka/streams/processor/TopicNameExtractor.html

I checked the release notes here and saw no note of any relevant change: https://www.apache.org/dist/kafka/2.2.0/RELEASE_NOTES.html

Answer 1

You have to distinguish between two cases: (1) Kafka Streams creates a topics, or (2) you have auto.create.topics.enable=true set on the brokers.

The docs cover the first case, ie, Kafka Streams will not issue a CreateTopic request to the brokers. Hence, the docs are correct.

However, if you have set auto.create.topics.enable=true on the brokers, the broker will create a non-existent topic if you try to read or write from/to it, or even if you only request metadata (like number of partitions) about it. I assume that you hit this case.

Btw: It's not recommended to enable auto topic creation on the brokers though, because this will create topics most likely will an unwanted number of partitions and it has a mayor semantic impact if you change the number of partitions later.