Increase number of partitions in a Kafka topic from a Kafka client

4
votes

I'm a new user of Apache Kafka and I'm still getting to know the internals.

In my use case, I need to increase the number of partitions of a topic dynamically from the Kafka Producer client.

I found other similar questions regarding increasing the partition size, but they utilize the zookeeper configuration. But my kafkaProducer has only the Kafka broker config, but not the zookeeper config.

Is there any way I can increase the number of partitions of a topic from the Producer side? I'm running Kafka version 0.10.0.0.

3
apache-kafka

3 Answers

7
votes

As of Kafka 0.10.0.1 (latest release): As Manav said it is not possible to increase the number of partitions from the Producer client.

Looking ahead (next releases): In an upcoming version of Kafka, clients will be able to perform some topic management actions, as outlined in KIP-4. A lot of the KIP-4 functionality is already completed and available in Kafka's trunk; the code in trunk as of today allows client to create and to delete topics. But unfortunately, for your use case, increasing the number of partitions is still not possible yet -- this is in scope for KIP-4 (see Alter Topics Request) but is not completed yet.

TL;DR: The next versions of Kafka will allow you to increase the number of partitions of a Kafka topic, but this functionality is not yet available.

2
votes

It is not possible to increase the number of partitions from the Producer client. Any specific use case use why you cannot use the broker to achieve this ?

But my kafkaProducer has only the Kafka broker config, but not the zookeeper config.

I don't think any client will let you change the broker config. You can only access (read) the server side config at max.

0
votes

Your producer can provide different keys for ProducerRecord's. The broker would place them in different partitions. For example, if you need two partitions, use keys "abc" and "xyz".

This can be done in version 0.9 as well.