Kafka auto partition

Question

Is there an initiative to implement topic auto partitioning such that partition logic is removed from the public api? I feel that Kafka cluster should have a dynamic partitioning adjusting to user message rate dynamically and storing individual records in one or n physical files and machines to maximize throughput and latency. I find that creating n partitions for a topic is equivalent to having n topics

Answer 1

According to the KIP page, no. And that makes sense. Topic partitioning is highly dependent on what exactly you're trying to achieve with the topic. It would be nearly impossible to create an algorithm that could handle all use cases.

Answer 2

In kafka message order is guaranteed in partition. But for different partitions in same topic there is no ordering guarantee. If you autoscale based on load and change partition of a message you cannot satisfy ordering guarantee. That's why there is no autoscaling for partitions in Kafka.