Kafka Topic Creation with --bootstrap-server gives timeout Exception (kafka version 2.5)

Question

When trying to create topic using --bootstrap-server, I am getting exception "Error while executing Kafka topic command: Timed out waiting for a node" :-

kafka-topics --bootstrap-server localhost:9092 --topic boottopic --replication-factor 3 --partitions

However following works fine, using --zookeeper :-

kafka-topics --zookeeper localhost:2181--topic boottopic --replication-factor 3 --partitions

I am using Kafka version 2.5 and as per knowledge since version >2.2, all the offsets and metadata are stored on the broker itself. So, while creating topic there's no need to connect to zookeeper.

Please help to understand this behaviour

Note - I have set up a Zookeeper quorum and Kafka broker cluster each containing 3 instance on a single machine (for dev purposes)

Answer 1

Old question, but Ill answer anyways for the sake of internet wisdom. You probably have auth set, when using --bootstrap-server you need to also specify your credentials with --command-config

Answer 2

since version >2.2, all the ... metadata are stored on the broker itself

False. Topic metadata is still stored on Zookeeper until KIP-500 is completed.

The AdminClient.createTopics() method, however that is used internally will delegate to Zookeeper from the Controller broker node in the cluster.

Hard to say what the error is, but most common issue is that Kafka is not running, you have SSL enabled and the certs are wrong, or the listeners are misconfigured.

For example, in the listeners, the default broker port on a Cloudera Kafka installation would be 6667, not 9092

each containing 3 instance on a single machine

Running 3 instances on one machine does not improve resiliency or performance unless you have 3 CPUs and 3 separate HDDs on that one motherboard.

Answer 3

"Error while executing Kafka topic command: Timed out waiting for a node"

This seems like your broker is down or is inaccessible from where you are running those commands or it hasn't started yet (perhaps still starting).

---

Sometimes the broker startup takes long because it performs some cleaning operations. You may want to check your Kafka broker startup logs and see if it is ready and then try creating the topics by giving in the bootstrap servers.

There could also be some errors during your Kafka broker startup like Too many open files or wrong zookeeper url, zookeeper not being accessible by your broker, to name a few.

---

If you are able to create topics by passing in your Zookeeper URL means that zookeeper is up but does not necessarily mean that your Kafka broker(s) are also up and running.

Since a zookeeper can start without a broker but not vice-versa.