Kafka Cluster (ZK, BR, BR, BR) in Vagrant can not establish connection

Question

The issue is that I am not able to send messages via the producer script to any of my broker inside a cluster.

The setup is a single zookeeper server that runs on the ip ( 192.168.10.2:2181) with default settings.

In addition, there are 3 brokers running on (192.168.10.101:9092,192.168.10.102:9092,192.168.10.103:9092).

After spinning up the broker I can see in the zookeeper shell that the 3 broker are connected.

vagrant ssh zookeeper-1
kafka/bin/zookeeper-shell.sh 192.168.10.2:2181
ls /brokers/ids

Output:

Connecting to 192.168.10.2:2181 Welcome to ZooKeeper! JLine support is disabled WATCHER:: WatchedEvent state:SyncConnected type:None path:null ls /brokers/ids [3, 2, 1]

Now I can create the topics with replication:

kafka/bin/kafka-topics.sh --create --zookeeper 192.168.10.2:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic

Afterwards I verify that the topic is created:

kafka/bin/kafka-topics.sh --describe --zookeeper 192.168.10.2:2181 --topic my-replicated-topic

Output:

Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs: Topic: my-replicated-topic Partition: 0 Leader: 3 Replicas: 3,1,2 Isr: 3,1,2

If I try now to send a message via the producer:

kafka/bin/kafka-console-producer.sh --broker-list 192.168.10.101:9092,192.168.10.102:9092,192.168.10.103:9092 --topic my-replicated-topic

[2017-12-28 13:53:30,530] WARN [Producer clientId=console-producer] Connection to node -3 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2017-12-28 13:53:30,536] WARN [Producer clientId=console-producer] Connection to node -2 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2017-12-28 13:53:30,539] WARN [Producer clientId=console-producer] Connection to node -1 could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

Same happens if I try to send messages from one of the broker boxes.

I uploaded the Vagrantfile + scripts here so you can reproduce my problem: https://github.com/0xBADBAC0N/kafka-vagrant-games

I basically googled now for hours and hope that anyone here maybe now how to fix that issue. I guess it is just a config setting that I fail to see.

Stack Overflow is a site for programming and development questions. This question appears to be off-topic because it is not about programming or development. See What topics can I ask about here in the Help Center. Perhaps Super User or Unix & Linux Stack Exchange would be a better place to ask. — jww
@jwwI agree with you after looking at your suggestions. After still digging into Kafka I found the solution. The broker, even if he should know his own address, needs to be defined in his own settings with '"listeners=PLAINTEXT://"$BROKER_IP":9092"'. This fixed all issues. :) — user3614590

user3614590 user3614590 · Accepted Answer · 2017-12-29T00:30:16

The broker, even if he should know his own address, needs to be defined in his own settings with '"listeners=PLAINTEXT://"$BROKER_IP":9092"'.

Kafka Cluster (ZK, BR, BR, BR) in Vagrant can not establish connection

1 Answers