Can't connect to kafka docker container from another docker container

Question

I'm trying to connect to kafka docker container from another docker container. But it didn't connect.

There are the list of containers using for kafka messaging

Network kafka-docker_default has two containers kafka-docker_zookeeper_1 and kafka-docker_kafka0_1

For running kafka and zookeeper I used docker-compose file:

version: '2'
services:
  zookeeper:
    image: "confluentinc/cp-zookeeper:latest"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000

# This has three listeners you can experiment with. 
# BOB for internal traffic on the Docker network
# FRED for traffic from the Docker-host machine (`localhost`)
# ALICE for traffic from outside, reaching the Docker host on the DNS name `never-gonna-give-you-up`
# Use 
  kafka0:
    image: "confluentinc/cp-kafka"
    ports:
      - '9092:9092'
      - '29094:29094'
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://kafka0:9092,LISTENER_ALICE://kafka0:29094
      KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092,LISTENER_FRED://localhost:9092,LISTENER_ALICE://never-gonna-give-you-up:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_BOB:PLAINTEXT,LISTENER_FRED:PLAINTEXT,LISTENER_ALICE:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_BOB
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1

Containers dockercompose17138956372294708100_kafkatest.producer_1 and dockercompose17138956372294708100_kafkatest.consumer_1 are for produce and consume messages. While container dockercompose17138956372294708100_kafkatest.producer_1 is trying to connect to kafka cluster using bootstrap.servers parameter(localhost:9092 or never-gonna-give-you-up:29094) it showed error

%7|1569945335.015|BROKERFAIL|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: failed: err: Local: Host resolution failure: (errno: Bad address)
%3|1569945335.015|FAIL|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Failed to resolve 'never-gonna-give-you-up:29094': Name or service not known (after 1656ms in state CONNECT)
%3|1569945335.015|ERROR|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Failed to resolve 'never-gonna-give-you-up:29094': Name or service not known (after 1656ms in state CONNECT)
%7|1569945335.015|STATE|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: never-gonna-give-you-up:29094/bootstrap: Broker changed state CONNECT -> DOWN
%3|1569945335.015|ERROR|rdkafka#producer-1| [thrd:never-gonna-give-you-up:29094/bootstrap]: 1/1 brokers are down

How can I fix it?

Answer 1

dockercompose17138956372294708100_kafkatest.producer_1 is trying to connect to kafka cluster using bootstrap.servers parameter(localhost:9092 or never-gonna-give-you-up:29094)

From another Docker container, neither of those are correct.

1. never-gonna-give-you-up does not exist anywhere as a Hostname/DNS record in your Docker configuration. Based on the error, your external DNS server also doesn't know what that is

2. localhost refers to the Kafka client container, not your host, or the broker container.

According to your compose configuration, you need to connect to kafka0 host/service, which is advertised on port 29092 for the bootstrap.servers

# BOB for internal traffic on the Docker network
KAFKA_ADVERTISED_LISTENERS: LISTENER_BOB://kafka0:29092

You should also probably remove LISTENER_ALICE because it isn't providing any useful connection details unless your host's external hostname is actually never-gonna-give-you-up

And you'll want to put your producer and consumer as part of your compose file

Answer 2

Please try this configuration:

version: '3'
services:
  zookeeper:
    image: confluentinc/cp-zookeeper:latest
    ports:
      - "32181:32181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 32181
      ZOOKEEPER_TICK_TIME: 2000

  kafka0:
    image: confluentinc/cp-kafka:latest
    ports:
     - "9090:9090"
    depends_on:
      - zookeeper
    environment:
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:32181
      KAFKA_ADVERTISED_LISTENERS: LISTENER_INTERNAL://kafka0:29090,LISTENER_EXTERNAL://localhost:9090
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: LISTENER_INTERNAL:PLAINTEXT,LISTENER_EXTERNAL:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: LISTENER_INTERNAL
      KAFKA_AUTO_CREATE_TOPICS_ENABLE: "false"

With this configuration, if you connect from host to the broker's container, you use the IP localhost:9090. If you connect from other containers with the same network, you use kafka0:29090.