Find broker id used in the Kafka cluster

4
votes

I want to know the list of taken broker ids in a kafka cluster. For example, in a cluster with 10 nodes if I create a topic with 10 partitions(or more) I can see from the output of a describe topic command, the brokers to which it has been assigned. 

./bin/kafka-topics --describe --zookeeper <zkconnect>:2181 --topic rbtest3

Can I collect this information without creating a topic?

6
apache-kafka

6 Answers

9
votes

You can get list of used broker ids using zookeeper cli.

zookeeper-3.4.8$ ./bin/zkCli.sh -server zookeeper-1:2181 ls /brokers/ids | tail -1
[0]
7
votes

You also can use the zookeeper-shell.sh script that ships with the Kafka distribution, like this:

linux$ ./zookeeper-shell.sh zookeeper-IPaddress:2181 <<< "ls /brokers/ids"

Just add the IP address of any of your Zookeeper servers (and/or change the port if necessary, for example when running multiple Zookeeper instances on the same server).

This alternative can be useful when, for example, you find yourself inside a container (Docker, LXC, etc.) that is exclusively running a Kafka client; but Zookeeper itself is somewhere else (say, in a different container).

I hope it helps. =:)

3
votes
#kafka broker id
cat $KAFKA_HOME/logs/meta.properties
1
votes

you can use kafka-manager, an open-source tool powered by yahoo.

1
votes

You can run the following command:

bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic=<your topic> --broker-list=<your broker list>  --time=-2

This will list all of the brokers with their id and the beginning offset.

1
votes

If you want to know what is the broker ID of a specific broker - the easiest way is to look at its controller.log, I found:

cat /var/log/kafka/controller.log
[2021-02-18 13:20:22,639] INFO [ControllerEventThread controllerId=1003] Starting (kafka.controller.ControllerEventManager$ControllerEventThread)
[2021-02-18 13:20:22,646] DEBUG [Controller id=1003] Broker 1002 has been elected as the controller, so stopping the election process. (kafka.controller.KafkaController)

controllerId=1003 ---> this is your brokerID (1003)

[substitute your path to the kafka logs, of course ...]