How to communicate with Kafka server running inside a docker

0
votes

I am using apache KafkaConsumer in my Scala app to talk to a Kafka server wherein the Kafka and Zookeeper services are running in a docker container on my VM (the scala app is also running on this VM). I have setup the KafkaConsumer's property "bootstrap.servers" to use 127.0.0.1:9092.

The KafkaConsumer does log, "Sending coordinator request for group queuemanager_testGroup to broker 127.0.0.1:9092". The problem appears to be that the Kafka client code is setting the coordinator values based on the response it receives which contains responseBody={error_code=0,coordinator={node_id=0,host=e7059f0f6580,port=9092}} , that is how it sets the host for future connections. Subsequently it complains that it is unable to resolve address: e7059f0f6580

The address e7059f0f6580 is the container ID of that docker container. I have tested using telnet that my VM is not detecting this as a hostname.

What setting do I need to change such that the Kafka on my docker returns localhost/127.0.0.1 as the host in its response ? Or is there something else that I am missing / doing incorrectly ?

1
networkingdockerapache-kafka
Well... the more relevant question is - "how are you running a containerised kafka service". Kafka needs zookeeper to work. And in general people use docker-compose to run the dependent containers. So... you will need to provide your Dockerfile, docker-compose.yaml or the commands for starting these containers. – sarveshseri
One more thing... please take some time to understand docker with something simpler such as a hello-world web application. And once you understand that you will understand the answer to this question. – sarveshseri

1 Answers

0
votes

Update

advertised.host.name is deprecated, and --override should be avoided.

Add/edit advertised.listeners to be the format of 

[PROTOCOL]://[EXTERNAL.HOST.NAME]:[PORT]

Also make sure that PORT is also listed in property for listeners

After investigating this problem for hours on end, found that there is a way to set the hostname while starting up the Kafka server, as follows: kafka-server-start.sh --override advertised.host.name=xxx (in my case: localhost)