What happens after modifying Kafka topic replication factor?

Question

According to the docs, you just modify a topic to increase the replication factor.

 > bin/kafka-topics.sh --zookeeper zk_host:port/chroot --create --topic my_topic_name 
       --partitions 20 --replication-factor 3 --config x=y

Unfortunately, it doesn't specify what happens then after you modify the topic. Do existing log segments get replicated to the new replicas, or only new messages?

This link is really helpful in answering your question stackoverflow.com/questions/46289511/… — Mohit Aggarwal

chaimt chaimt · Accepted Answer · 2017-01-03T18:11:14

For kafka 10 you need to invoke the kafka-reassign-partitions.sh script to change the replication factor of the topic. Here is a demo of a script that will display the topic before and after the change:

updateTopicReplication() {
    TOPIC_NAME=$1
    REPLICAS=$2

    echo "****************************************"
    echo "describe $TOPIC_NAME"
    bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC_NAME

    JSON_FILE=./configChanges/${TOPIC_NAME}-update.json
    echo $JSON_FILE
    [ -e $JSON_FILE ] && rm $JSON_FILE

    touch $JSON_FILE

    echo -e "{\"version\":1, \"partitions\":[{\"topic\":\"${TOPIC_NAME}\",\"partition\":0,\"replicas\":[${REPLICAS}]}]}" >> $JSON_FILE

    echo "****************************************"
    echo "updateing $TOPIC_NAME"
    bin/kafka-reassign-partitions.sh --zookeeper localhost:2181 --reassignment-json-file $JSON_FILE --execute

    echo "****************************************"
    echo "describe $TOPIC_NAME"
    bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic $TOPIC_NAME
}

What happens after modifying Kafka topic replication factor?

2 Answers