Kafka automatic replication to new node

0
votes

I am a newbie to Kafka and wanted to understand the behaviour of apache Kafka in below scenario.

Consider I have the topic with 3:

  • partitions 3
  • brokers 3
  • replication factor 3
  • min isr 2
  • Producer acks = all
  • Unclean leader election false

As per my understanding if broker 1 goes down there is no harm and no data loss as isr=2 and writes will be successful.

If node 1 comes back up it will again follow the leader and catch up.

My question is if node 1 never comes back up..also it's removed from isr list... My replication factor desired is 3... and if I add new node 4 ..how to automatically make node 4 copy the partitions from failed node 1...so that replication of 3 is still maintained?

1
apache-kafkaconfluent-platform
If node 1 is offline, and you add node 4, then any data from node 1 wouldn't be able to be copied – OneCricketeer

1 Answers

1
votes

Topic replicas stored in zookeper and if you add new broker you should update replicas value for topic(s). (As far as I know there is no way to do it automatically)

But you can do it manually by using kafka-reassign-partitions.sh tool.

Steps:

  • Create a json file to represent your desired replicas for partitions. For example:
{"version":1,
  "partitions":[
     {"topic":"YourTopic","partition":0,"replicas":[3,2,4]},
     {"topic":"YourTopic","partition":1,"replicas":[2,4,3]},
     {"topic":"YourTopic","partition":2,"replicas":[4,3,2]}
]}

  • Execute this command to reassign partitions.

    ./kafka/bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file my_file.json --execute