How to add a new Kafka brokers machines dynamically to a cluster

Question

We have Kafka confluent cluster , cluster include 3 Kafka brokers ,

Version details:

Kafka machines are installed on rhel version 7.2 
Kafka confluent version is 0.1x
Zookeeper version: 3.4.10
schema-registry version: 4.0.0

Each Kafka broker machine include the following services

Kafka broker
Zookeeper server
Schema registry

Now we want to add a additionally new 3 Kafka brokers machines to the current Kafka cluster ( the Additional Kafka machines are – kafka04/05/06 with the same Kafka version – 0.1X , )

So cluster should be finally with:

6 Kafka brokers machines - kafka01, kafka02 , kafka03 , kafka04 , kakfa05 , kafka05

3 zookeeper servers - kafka01, kafka02 , kafka03

3 schema registry services - kafka01, kafka02 , kafka03

In order to connect the new 3 Kafka brokers to the existing Kafka cluster we need to change the Configuration on all Kafka Machines ( the old Kafka machines and the new Kafka machines )

We are not sure what are exactly the configuration files in Kafka brokers that should be changed, But From my understanding We should change the Kafka and zookeeper settings as the following

I will happy to get remarks/notes about the following procedure

Edit the server.properties file on the new Kafka brokers - kafka04/05/06 , and change the broker.id parameter as the following

On kafka04 - broker.id=4 On kafka05 - broker.id=5 On kafka06 - broker.id=6
Edit the server.properties on all Kafka machines - kafka01/02/03/04/05/06 And change the following parameters to the number of total nodes in the cluster
```
 offsets.topic.replication.factor=6
 transaction.state.log.replication.factor=6
```
On the new Kafka machines - Kafka 04/05/06 edit the server.properties and update the parameter -- zookeeper.connect with the zookeeper server ip’s that located on kafka01 , kafka02 , kafka03

Example
```
 zookeeper.connect=10.10.10.1:2181,10.10.10.2:2181,10.10.10.3:2181
```

On Kafka machines – kafka 04/05/06 , edit the file - zookeeper.properties as the
Following

server.1=10.10.10.1:2888:3888
server.2=10.10.10.2:2888:3888
server.3=10.10.10.3:2888:3888

Edit the file - myid file on Kafka 04/05/06 , and change the parameter broker.id as the Following

on kafka04 set:
```
 broker.id=4
```
on kafka05 set:
```
 broker.id=5
```
on kafka06 set:
```
 broker.id=6
```
After settings as above , restart all Kafka brokers services on kafka01/02/03/04/05/06 and Restart the zookeeper servers on kafka01/02/03

And verify all Kafka services and zookeeper service started successfully

Reference info - https://www2.microstrategy.com/producthelp/current/PlatformAnalytics/en-us/Content/Add_kafka_node_to_kafka_cluster.htm

Mickael Maison Mickael Maison · Accepted Answer · 2020-08-05T20:31:58

When adding brokers, you don't need to change the configuration of existing brokers, nor restart them. The same applies to ZooKeeper, if you are not adding new ZooKeeper servers.

On the new brokers, you just need to set a different broker.id value.

I don't recommend increasing the replication factor of the topics (including internal) beyond 3.

Once your new brokers are started, you may want to rebalance your existing data. There are many tools to do that, including the kafka-reassign-partitions.sh tool. The Kafka docs has a section detailing the process: https://kafka.apache.org/documentation/#basic_ops_cluster_expansion

How to add a new Kafka brokers machines dynamically to a cluster

1 Answers