How to retrieve details of all broker from kafka cluster

Question

How should we retrieve the full details of all brokers(connected/disconnected) from kafka cluster/zookeeper ?

I found following way to fetch only active brokers, but I want the the IP address of broker which is serving previously in cluster but it is disconnected now

Following code snippet gives list of active brokers:

ZooKeeper zkInstance = new ZooKeeper("mymachine:port", 10000, null);
brokerIDs = zkInstance.getChildren("/brokers/ids", false);
for (String brokerID : brokerIDs) {
    brokerInfo = new String(zkInstance.getData("/brokers/ids/" + brokerID,     false, null));
    String     host=brokerInfo.substring(brokerInfo.indexOf("\"host\"")).split(",")    [0].replaceAll("\"","").split(":")[1];
    String     port=brokerInfo.substring(brokerInfo.indexOf("\"jmx_port\"")).split(",")    [0].replaceAll("\"","").split(":")[1];
    System.out.println(host+":"+port);              
}

Output:

my-machine-1:port
my-machine-2:port
my-machine-3:port
my-machine-4:port

I need information of all connected/disconnected brokers in multi node kafka cluster

If it's disconnected, I'm not sure how you're expecting to get that information... You should be monitoring JMX directly on the Kafka servers to determine what they're connected to — OneCricketeer
Kafka allocates some amount of replicas to partitions and these partitions are used by the topics, replicas are the actual nodes in cluster so there might be the provision given in Kafka to store the meta-data for replicas like broker number, JMX port and broker port where process is running, while these replicas are active ZooKeeper allows us to fetch these details but not in the case when broker goes down/inactive. — MAS
That's assuming you have replicas for a topic, though. Plus, if a replica is down, though, then it'll have under-replicated partitions and list a -1 when you describe the topic — OneCricketeer
Yes I got the same output when I described that particular topic, As you use the term under-replicated partitions so what kind information exactly it will contains? Is it only contains broker id and its status or it will contains all the metadata for that particular replica. — MAS

Manjunatha H C Manjunatha H C · Accepted Answer · 2018-11-12T12:20:23

Use describeCluster() of the AdminClient class to get the broker details such as host,port,id and rock.

Please refer the following code:

Properties kafkaProperties = new Properties();
kafkaProperties.put("bootstrap.servers", "localhost:9092,localhost:9093,localhost:9094");
AdminClient adminClient = AdminClient.create(kafkaProperties);
DescribeClusterResult describeClusterResult = adminClient.describeCluster();
Collection<Node> brokerDetails = describeClusterResult.nodes().get();
System.out.println("host and port details");
for(Node broker:brokerDetails) {
    System.out.println(broker.host()+":"+broker.port());
}

How to retrieve details of all broker from kafka cluster

1 Answers