Docker: multi kafka brokers failed

Question

I would like to use docker with several kafka brokers.

So I started some tests with this docker image => https://github.com/ches/docker-kafka

For one broker all work fine:

I start my zookeeper:

docker run -d --name zookeeper jplock/zookeeper:3.4.6

Then I start a kafka instance:

docker run -d --name kafka --link zookeeper:zookeeper ches/kafka

When I create topic and messages, all work.

Now I create a second kafka instance:

docker run -d --name kafka2 --link zookeeper:zookeeper --expose 9093  --env-file env ches/kafka

I change the exposed port to 9093 and environment variables:

PORT=9093
EXPOSED_PORT=9093
BROKER_ID=2

The broker starts well and I can create a topic with replication:

docker -D run --rm ches/kafka kafka-topics.sh --create --topic test2  --replication-factor 2 --partitions 1 --zookeeper $ZK_IP:2181

Now when I want to send some messages:

docker run --rm --interactive ches/kafka kafka-console-producer.sh --topic test2 --broker-list $KAFKA_IP:9092

I get this error:

ERROR Producer connection to 172.17.0.17:9093 unsuccessful (kafka.producer.SyncProducer) java.net.ConnectException: Connection refused

A docker ps give me that:

CONTAINER ID        IMAGE                    COMMAND                CREATED              STATUS              PORTS                          NAMES
d7bba0f3d0db        ches/kafka:latest        "kafka-console-produ   About a minute ago   Up About a minute   9092/tcp, 7203/tcp             sick_shockley
9c475a659383        ches/kafka:latest        "/start.sh"            4 minutes ago        Up 4 minutes        7203/tcp, 9092/tcp, 9093/tcp   kafka2
63aae4c539ab        ches/kafka:latest        "/start.sh"            28 minutes ago       Up 28 minutes       7203/tcp, 9092/tcp             kafka
ab560690e0e7        jplock/zookeeper:3.4.6   "/opt/zookeeper/bin/   28 minutes ago       Up 28 minutes       2181/tcp, 2888/tcp, 3888/tcp   zookeeper

So kafka2 seems started on 9093 port.

Why I get this error ?

Thanks.

C4stor C4stor · Accepted Answer · 2015-07-01T20:42:38

Check in the zookeeper docker what is the advertised host name of the kafka dockers. It's very possible that they registered their docker hash as host name (as it's the result of getInetAdress() within a docker container) instead of a resolvable address.

If that's the case, editing your standard kafka config to change advertised.host.name should solve your problem (it's a bit annoying because you have to change it at start, but you can for example fetch it from the /etc/hosts file of the docker container at startup, it should be the first half of the first line in it).

Docker: multi kafka brokers failed

1 Answers