Unable to run console consumer using a Kafka broker inside a Docker container

Question

I'm having a problem running a Kafka broker in a Docker container.

I have downloaded and unpacked a tar archive of Kafka 2.12-2.4.1.

I can run Zookeeper and Kafka brokers from the command line, producing and consuming messages successfully. When I build a docker container with Zookeeper, I can run that with a Kafka broker running from the command line, again producing and consuming messages successfully. When I put the broker into a docker container, the consumers and producers can't find a broker to connect to.

I have checked to ensure that Zookeeper has a connected broker: the zookeeper "dump" command below

echo dump | nc localhost 2181

returns

SessionTracker dump:
Session Sets (3)/(1):
0 expire at Thu May 28 09:53:33 GMT 2020:
0 expire at Thu May 28 09:53:36 GMT 2020:
1 expire at Thu May 28 09:53:39 GMT 2020:
    0x10002dfed700008
ephemeral nodes dump:
Sessions with Ephemerals (1):
0x10002dfed700008:
    /controller
    /brokers/ids/0
Connections dump:
Connections Sets (2)/(2):
0 expire at Thu May 28 09:53:37 GMT 2020:
2 expire at Thu May 28 09:53:47 GMT 2020:
    ip: /172.18.0.1:55656 sessionId: 0x0
    ip: /172.18.0.5:48474 sessionId: 0x10002dfed700008

which, to my untrained eye, looks like Zookeeper has a registered broker.

I can successfully list topics with the command

kafka-topics.sh --list --zookeeper localhost:2181

and when I run the command

netstat -plnt

I can see that I have listeners on ports 2181 and 9092, as below

tcp6       0      0 :::9092                 :::*                    LISTEN      54661/docker-proxy
tcp6       0      0 :::2181                 :::*                    LISTEN      47872/docker-proxy

However, when I try to connect a consumer with the command

kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic aaaa

I get errors

WARN [Consumer clientId=consumer-console-consumer-8228-1, groupId=console-consumer-8228] Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected (org.apache.kafka.clients.NetworkClient)

which would seem to indicate that the client can't contact the broker.

I've tried several different variations on the listeners and advertised.listeners in the broker config file, setting them to localhost and 127.0.0.1, but to no avail.

I'm sure that I'm missing something simple: can anyone help?

It is not very clear, do you have 2 containers and you try to connect from one to another using localhost as address? — leopal
Are you building your own Docker containers for your own learning? Because there are very good ones that exist already if all you want to do is run Kafka + Zookeeper on Docker: hub.docker.com/r/confluentinc/cp-kafka — Robin Moffatt
Also where you are running you Consumer from? Your host machine, or another Docker container? — Robin Moffatt
I have two containers and I'm trying to connect from one to another, I am doing this as a learning exercise, and I'm running the consumer from my host machine - although I tried to run it from another Docker container, with the same results. — Graeme Burton
To explain a little further, I have two containers, one running Zookeeper and the other running a broker. — Graeme Burton

Giorgos Myrianthous Giorgos Myrianthous · Accepted Answer · 2020-05-28T10:43:01

Make sure to configure the following:

KAFKA_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://kafka-host:9092
KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka-host:29092,LISTENER_EXTERNAL://localhost:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL

Now all your clients that run within the Docker Network, should use
```
LISTENER_INTERNAL://kafka-host:29092
```
All clients outside the Docker Network should use
```
LISTENER_EXTERNAL://kafka-host:9092
```

For a more comprehensive guide, read @RobinMoffatt article.

Unable to run console consumer using a Kafka broker inside a Docker container

2 Answers