kafka + description of the leaders and replications one of the leader not exists

Question

we have ( 3 kafka machines )

kafka brokers id's - are ( 101 , 102 ,103 )

ambari version - 2.6.1

hadoop version 2.6.5

29 datanode machines

3 journal machines ( master machines )

when we run the following command

kafka-topics.sh --zookeeper 10.231.23.45:2181 –describe

we see the description of the leaders and replications

but what is wrong here is we see only two leaders for each topic

in our case the topic name - MLK_PLW_QA , but the same situation is on all the topics

Topic:MLK_PLW_QA    PartitionCount:100  ReplicationFactor:3 Configs:retention.bytes=9239504949
    Topic: MLK_PLW_QA   Partition: 0    Leader: 102 Replicas: 102,101,103   Isr: 101,102,103
    Topic: MLK_PLW_QA   Partition: 1    Leader: 102 Replicas: 103,102,101   Isr: 101,102
    Topic: MLK_PLW_QA   Partition: 2    Leader: 101 Replicas: 101,103,102   Isr: 101,102,103
    Topic: MLK_PLW_QA   Partition: 3    Leader: 102 Replicas: 102,103,101   Isr: 101,102,103
    Topic: MLK_PLW_QA   Partition: 4    Leader: 101 Replicas: 103,101,102   Isr: 101,102
    Topic: MLK_PLW_QA   Partition: 5    Leader: 101 Replicas: 101,102,103   Isr: 101,102,103
    Topic: MLK_PLW_QA   Partition: 6    Leader: 102 Replicas: 102,101,103   Isr: 101,102,103
    Topic: MLK_PLW_QA   Partition: 7    Leader: 102 Replicas: 103,102,101   Isr: 101,102

I must to say that this situation not appears before couple days , and we see all leaders 101 , 102 , 103 ( from kafka-topics.sh --zookeeper 10.231.23.45:2181 –describe )

what are the reasons that broker id - 103 not appears as leader ?

note - all kafka brokers services are running ( we can see that from ambari gui )

Steephen Steephen · Accepted Answer · 2019-01-17T04:11:46

One possibility, node with broker id 103 may be crashed/stopped or was down for a while. So cluster did a rebalance and selected the new leader from available nodes. Even though now broker 103 is alive, cluster controller didn't change the leadership of topics to optimize the computing.

kafka + description of the leaders and replications one of the leader not exists

1 Answers