KAFKA - how to re-balance the partitions to available brokers

Question

we have 3 kafka machines and 3 zookeeper's servers

kafka1 - 1001 ( broker ID )
kafka2 - 1002 ( broker ID )
kafka3 - 1003 ( broker ID )

we have issue is about to re-balance the partitions to available brokers , we can see ( down ) that some partitions under-replicated with only two brokers in isr. Instead with 3

what is the best way to re balance the kafka topic partitions in ISR

second - we can see that leader 1002 is missing , what is the solution for this?

remark - we have 23 topics ( the list down is partial )

[kafka@kafka01 bin]$ ./kafka-topics.sh -describe --zookeeper master:2181
Topic:lop_gt        PartitionCount:50       ReplicationFactor:3     Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
        Topic: lop_gt       Partition: 0    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 6    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 12   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 18   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 24   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 30   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 36   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 42   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: lop_gt       Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: lop_gt       Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: lop_gt       Partition: 48   Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: lop_gt       Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
Topic:_schemas  PartitionCount:1        ReplicationFactor:3     Configs:cleanup.policy=compact
        Topic: _schemas Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
Topic:ambari_kafka_service_check        PartitionCount:1        ReplicationFactor:1     Configs:
        Topic: ambari_kafka_service_check       Partition: 0    Leader: 1002    Replicas: 1002  Isr: 1002
Topic:jr_dfse    PartitionCount:10       ReplicationFactor:3     Configs:
        Topic: jr_dfse   Partition: 0    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 1    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 2    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 3    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: jr_dfse   Partition: 4    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 5    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 6    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: jr_dfse   Partition: 7    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: jr_dfse   Partition: 8    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: jr_dfse   Partition: 9    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:frte_nnc   PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: frte_nnc  Partition: 0    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 1    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: frte_nnc  Partition: 2    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: frte_nnc  Partition: 3    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: frte_nnc  Partition: 4    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: frte_nnc  Partition: 5    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
Topic:erw_plk    PartitionCount:100      ReplicationFactor:3     Configs:
        Topic: erw_plk   Partition: 0    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 1    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 2    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 3    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 4    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 5    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 6    Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 7    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 8    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 9    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 10   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 11   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 12   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 13   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 14   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 15   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 16   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 17   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 18   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 19   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 20   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 21   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 22   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 23   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 24   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 25   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 26   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 27   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 28   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 29   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 30   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 31   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 32   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 33   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 34   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 35   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 36   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 37   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 38   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 39   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 40   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 41   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 42   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 43   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 44   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 45   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 46   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 47   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 48   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 49   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 50   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 51   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 52   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 53   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 54   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 55   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 56   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 57   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 58   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 59   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 60   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 61   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 62   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 63   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 64   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 65   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 66   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 67   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 68   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 69   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 70   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 71   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 72   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 73   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 74   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 75   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 76   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 77   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 78   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 79   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 80   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 81   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 82   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 83   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 84   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 85   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 86   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 87   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 88   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 89   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 90   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 91   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 92   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 93   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: erw_plk   Partition: 94   Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 95   Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 96   Leader: 1003    Replicas: 1002,1003,1001        Isr: 1003,1001
        Topic: erw_plk   Partition: 97   Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: erw_plk   Partition: 98   Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002
        Topic: erw_plk   Partition: 99   Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
Topic:loe_sd        PartitionCount:6        ReplicationFactor:3     Configs:
        Topic: loe_sd       Partition: 0    Leader: 1001    Replicas: 1002,1001,1003        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 1    Leader: 1003    Replicas: 1003,1002,1001        Isr: 1003,1001
        Topic: loe_sd       Partition: 2    Leader: 1001    Replicas: 1001,1003,1002        Isr: 1003,1001,1002
        Topic: loe_sd       Partition: 3    Leader: 1001    Replicas: 1002,1003,1001        Isr: 1001,1003,1002
        Topic: loe_sd       Partition: 4    Leader: 1003    Replicas: 1003,1001,1002        Isr: 1003,1001
        Topic: loe_sd       Partition: 5    Leader: 1001    Replicas: 1001,1002,1003        Isr: 1003,1001,1002

still not , but what is your suggestion about what to search in the logs , do you mean thta we need to focus about network problems or else? — King David
I not see something un-usual in the logs , can you please advice what are the reasons for this issue , so according to that , I will focus the searching — King David
Well, you setup the cluster using Ambari and the servers should be all in the same LAN, so if you did have network issues, then there is just something wrong with how you prepared the servers to start with. One better explanation could be that some partitions are sitting on a failed hard drive Anyway, clearly one of your servers is not healthy and the logs or some other JMX metrics from the Kafka process should be indicating there is something wrong with it (hint: install and setup Grafana in HDP to monitor Kafka) — OneCricketeer
we also checked the network and the disk that is mounted , and they are fine , so we need to search other things , see jakub answer he said to "more time to re-sync the data" do you know what is mean ? — King David

Jakub Jakub · Accepted Answer · 2018-09-29T20:59:51

You have two options how to influence the partition leadership. With the configuration option auto.leader.rebalance.enable set to true (which should be the default setting), Kafka will automatically try to move the leadership for each partitionto your preferred broker. Preferred broker is the first one in the list of replicas. It is executed as periodical check, so might not happen immediately.

Alternatively - if the automatic rebalance is turned off - you can reassign the replicas manually using the bin/kafka-preferred-replica-election.sh tool. For more info see the Kafka docs.

However, in your case it looks like your broker 1002 is either shortly after the restart or might need more time to re-sync the data, since the broker 1002 is not yet ISR for all partitions. If this is permanent state and you are 100% sure that your broker had enough time to sync all partitions etc., maybe there are some other problems with 1002. But that is hard to say without logs from the broker etc.

KAFKA - how to re-balance the partitions to available brokers

1 Answers