Unable to connect to Zookeeper on external server via docker

Question

I'm currently new to all of these tools and request you to bear with me kindly.

I'm trying to create a data pipeline to push the changes of a MySQL DB to a Kafka Broker. I'm loosely trying to follow the tutorial here. If all the docker containers (directly downloaded from the debezium repos) are located inside the local machine, I'm able to get the system running.

However, if I place the MySQL, Kafka and Zookeeper dockers on another machine on my local network and try to connect it I get the following error:

docker: Error response from daemon: could not get container for zookeeper: No such container: zookeeper.

Following is the command I'm using to achieve this:

sudo docker run -it --rm --name connect \
    -p 8083:8083 \
    -e GROUP_ID=1 \
    -e CONFIG_STORAGE_TOPIC=my_connect_configs \
    -e OFFSET_STORAGE_TOPIC=my_connect_offsets \
    -e ADVERTISED_HOST_NAME=192.168.31.214 \
    -e ADVERTISED_PORT=2181 \
    --link zookeeper:zookeeper \
    --link kafka:kafka \
    --link mysql:mysql \
    debezium/connect:0.8

The two machines can communicate with each other over all the ports that are involved (2181 for Zookeeper, 9092 for Kafka and 3306 for MySQL).

I have also tried providing the property name

-e ADVERTISED_PORT=2181

and modifying the host property to

-e ADVERTISED_HOST_NAME=192.168.31.214:2181

both of which didn't help.

As per my understanding, there is some property to bind individual links to ports on the machine, but I am unable to find the said property. Please let me know where I'm going wrong with this.

OneCricketeer OneCricketeer · Accepted Answer · 2018-08-09T00:18:09

I do not think -e ADVERTISED_PORT=2181 makes sense for the Debezium container. That's a Zookeeper port, which is running elsewhere, not inside Debezium. Similarly for the advertised hostname, that's a Kafka server property, not being loaded by Debezium container (and it should not be using port 2181 either)

If Zookeeper and the others are on another machine then --link won't work. That tutorial assumes everything runs on one machine

If you're using Docker Swarm, you need to setup and use an "overlay network".

Otherwise, you need point your Debezium configuration at the external IP address of the other machine. However, this is not "the Docker way" to solve the problem as you shouldn't have to know which hosts your containers are running on.

Unable to connect to Zookeeper on external server via docker

1 Answers