Docker Error : kafka.zookeeper.ZooKeeperClientTimeoutException in Kafka shell

Question

I'm trying to run kafka using docker.

I started kafka service.

docker-compose ps

     Name                        Command               State                         Ports

kafka-docker_kafka_1 start-kafka.sh Up 0.0.0.0:9092->9092/tcp
kafka-docker_zookeeper_1 /bin/sh -c /usr/sbin/sshd ... Up 0.0.0.0:2181->2181/tcp, 22/tcp, 2888/tcp, 3888/tcp

docker-compose.yml

cat docker-compose.yml

version: '2'
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    build: .
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: kafka
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 36000
      KAFKA_ADVERTISED_PORT: 9092
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

When I tried to create a topic by entering inside shell, it's throwing the below error.

./start-kafka-shell.sh kafka zookeeper:2181

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $ZK --replication-factor 1

[2019-02-12 08:07:59,097] WARN Session 0x0 for server zookeeper:2181, unexpected error, closing socket connection and attempting reconnect (org.apache.zookeeper.ClientCnxn)
java.nio.channels.UnresolvedAddressException
    at sun.nio.ch.Net.checkAddress(Net.java:101)
    at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:622)
    at org.apache.zookeeper.ClientCnxnSocketNIO.registerAndConnect(ClientCnxnSocketNIO.java:277)
    at org.apache.zookeeper.ClientCnxnSocketNIO.connect(ClientCnxnSocketNIO.java:287)
    at org.apache.zookeeper.ClientCnxn$SendThread.startConnect(ClientCnxn.java:1021)
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1064)

Tried 2 possibilities:

Got timeout exception on changing hostname :zookeeper(though I mentioned the same in docker-compose.yml ) to ip: 0.0.0.0

./start-kafka-shell.sh kafka 0.0.0.0:2181

bash-4.4# $KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $ZK --replication-factor 1

Exception in thread "main" kafka.zookeeper.ZooKeeperClientTimeoutException: Timed out waiting for connection while in state: CONNECTING
    at kafka.zookeeper.ZooKeeperClient.$anonfun$waitUntilConnected$3(ZooKeeperClient.scala:268)
    at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:12)
    at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:251)
    at kafka.zookeeper.ZooKeeperClient.waitUntilConnected(ZooKeeperClient.scala:264)
    at kafka.zookeeper.ZooKeeperClient.<init>(ZooKeeperClient.scala:97)
    at kafka.zk.KafkaZkClient$.apply(KafkaZkClient.scala:1694)
    at kafka.admin.TopicCommand$.main(TopicCommand.scala:57)
    at kafka.admin.TopicCommand.main(TopicCommand.scala)

Inorder to avoid the above error, I increased the timeout parameter from 6000 to 36000 in docker-compose.yml --> KAFKA_ZOOKEEPER_CONNECTION_TIMEOUT_MS: 36000

I'm getting the same timeout error.

you should introduce host address and port of the Zookeeper for the property --zookeeper. Suppose the zookeeper process is running on localhost: $KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper localhost:2181 --replication-factor 1 — Soheil Pourbafrani
You can find a functioning Docker Compose here: github.com/confluentinc/cp-docker-images/blob/5.1.0-post/… You can find important information about networking in Docker with Kafka here: rmoff.net/2018/08/02/kafka-listeners-explained — Robin Moffatt
@Soheil : I have already given hostname and port while starting the shell. It's mentioned in the post.Tried with zookeeper:2181 and 0.0.0.0:2181. One is giving unresolved address resolution exception and other is giving timeout exception respectively. — ISM
Resolved.Answer:- Until you explicitly add --net kafka-docker_default to the start-kafka-shell.sh script, the internal docker DNS resolution for host zookeeper won't work, so further examples about producer/consumer cannot be run. The correct script should look like this:#!/bin/bash docker run --rm -v /var/run/docker.sock:/var/run/docker.sock --net kafka-docker_default -e HOST_IP=$1 -e ZK=$2 -i -t wurstmeister/kafka /bin/bash — ISM
I've rolled back your edit. It is not proper here to add RESOLVED or SOLVED to the title, and it is also not proper to edit the solution into the question. Instead, you should write an answer in the space below that is designed for that purpose; see Can I answer my own question? for more information. If you don't want to write an answer, then you can delete the question using the link beneath the tags or wait to see if someone else posts an answer.. — Ken White

OneCricketeer OneCricketeer · Accepted Answer · 2019-02-13T03:56:21

There are lots of pre-built Kafka images. It's hard to tell what you're actually trying to do, but seems you're rebuilding an image? Or building your own?

You definitely don't need the zookeeper address as a parameter to start Kafka, though

That being said, 0.0.0.0 is not a real address to connect to. It's often used as a bind address meaning to "allow all connections to this server"

Then zookeeper service name is only resolvable within the Docker network, so that might explain why you need --net, but that implies you're using docker run for something, which again isn't clear given you're using Compose

If you want to get into a shell of the container, you should just be to use docker-compose exec kafka bash, and Zookeeper is already an environment variable of that container

So something like

docker-compose exec kafka bash -c '$KAFKA_HOME/bin/kafka-topics.sh --create --topic sentiment --partitions 1 --zookeeper $KAFKA_ZOOKEEPER_CONNECT --replication-factor 1'

Docker Error : kafka.zookeeper.ZooKeeperClientTimeoutException in Kafka shell

3 Answers