Kafka - Error creating broker listeners between Linux and Windows

Question

We are trying to message from Linux to windows using apache Kafka; Zookeeper and Kafka is configured in both OS and messaging happening in respective localhost;

Error while messaging from windows to Linux;

error - ERROR Error when sending message to topic my_topic with key: null, value: 5 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Topic my_topic not present in metadata after 60000 ms.

[2020-10-09 11:24:22,903] WARN [Producer clientId=console-producer] Error while fetching metadata with correlation id 51 : {my_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) Terminate batch job (Y/N)?

[20-10-09 11:24:28,979] WARN [Consumer clientId=consumer-1, groupId=console-consumer-56715] Error while fetching metadata with correlation id 81 : {my_topic=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient) Processed a total of 0 messages

Listed below are the configuration files for Linux server endpoint 
    
#producer properties
metadata.broker.list=localhost:9092,192.168.1.3:9092,192.168.0.103:9092 

#consumer properties
zookeeper.connect=127.0.0.1:2181,192.168.1.3:2181,192.168.0.103:2181
#consumer group id
group.id=test-consumer-group


#server properties
broker.id=1
listeners=PLAINTEXT://localhost:9092
advertised.listeners=PLAINTEXT://192.168.1.3:9092,192.168.0.103:9092
zookeeper.connect=localhost:2181


#192.168.0.103 is windows machine
#172.22.3.28 is Linux machine

Windows producer – 
kafka-console-producer.bat --broker-list 172.22.3.28:9092 --topic my_topic

Linux Consumer-
bin/kafka-console-consumer.sh --new-consumer --bootstrap-server 192.168.0.103:9092 --topic my_topic --from-beginning

we started configuring kafka individual in linux and windows and successfully send message thru localhost. presume real-time messaging across OS is possible with kafka. please correct.

Robin Moffatt Robin Moffatt · Accepted Answer · 2020-10-12T10:14:03

You need to configure the advertised.listeners such that the Windows client can correctly connect to the Linux broker.

If the Linux broker has IP 172.22.3.28 then the config should look like:

listeners=PLAINTEXT://0.0.0.0:9092
advertised.listeners=PLAINTEXT://172.22.3.28:9092

Ref: https://www.confluent.io/blog/kafka-client-cannot-connect-to-broker-on-aws-on-docker-etc

Kafka - Error creating broker listeners between Linux and Windows

1 Answers