kafka: cannot produce messages to kafka inside docker

Question

docker-compose.yml (https://github.com/wurstmeister/kafka-docker)

version: "2.1"
services:
  zookeeper:
    image: wurstmeister/zookeeper
    ports:
      - "2181:2181"
  kafka:
    image: wurstmeister/kafka
    ports:
      - "9092:9092"
    environment:
      KAFKA_ADVERTISED_HOST_NAME: localhost
      KAFKA_ADVERTISED_PORT: 9092
      KAFKA_CREATE_TOPICS: "test:3:1"
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

Errors when trying to produce messages following https://kafka.apache.org/quickstart:

~/kafka_2.11-1.0.0$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>gh 
>[2018-01-19 17:28:15,385] ERROR Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1566 ms has passed since batch creation plus linger time

list topics:

~/kafka_2.11-1.0.0$ bin/kafka-topics.sh --list --zookeeper localhost:2181
__consumer_offsets
test

why? thanks

UPDATE

how to set KAFKA_ADVERTISED_HOST_NAME or network to make my python/java program or kafka-console-producer.sh (outside docker container) to produce messages to the kafka by localhost:9092?

UPDATE

It seems that the following docker-compose.yml working fine

version: "2"

services:
  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    network_mode: "host"
    ports:
      - 2181:2181
  kafkaserver:
    image: "wurstmeister/kafka:latest"
    network_mode: "host"
    ports:
      - 9092:9092
    environment:
      KAFKA_CREATE_TOPICS: "test:3:1"
      KAFKA_ZOOKEEPER_CONNECT: localhost:2181

i guess you bing kafka service to localhost, so you can access kafka only in kafka container. — Shen Yudong
You might want to look at the Kafka Docker quickstart guide by Confluent — OneCricketeer

Bertrand_Szoghy Bertrand_Szoghy · Accepted Answer · 2018-05-06T15:49:18

I had the same issue. The suggested syntax in the kafka-docker README does not match the provided docker-compose.yml which does not work as is. I finally found this post and a variation of BEA's updated docker-compose.yml file worked for me. Thank you!

Here are the details.

I am running wurstmeister/kafka-docker on a Ubuntu 16.04 virtual image I set up as described at https://bertrandszoghy.wordpress.com/2018/05/03/building-the-hyperledger-fabric-vm-and-docker-images-version-1-1-from-scratch/

My docker-compose.yml file:

version: '2'
services:
  zookeeper:
    image: "wurstmeister/zookeeper:latest"
    network_mode: "host"
    ports:
      - "2181:2181"
  kafka:
    image: "wurstmeister/kafka:latest"
    network_mode: "host"
    ports:
      - 9092:9092
    environment:
      KAFKA_LISTENERS: PLAINTEXT://:9092
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://172.17.0.1:9092
      KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
      KAFKA_CREATE_TOPICS: "BertTopic:3:1"

On the same VM I installed NodeJs with:

curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash –
sudo apt-get install -y nodejs
cd
mkdir nodecode
cd nodecode
sudo npm install -g node-pre-gyp
sudo npm install kafka-node

Then I ran the following program to produce a couple of messages:

var kafka = require('kafka-node'),
    Producer = kafka.Producer,
    KeyedMessage = kafka.KeyedMessage,
    client = new kafka.Client(),
    producer = new Producer(client),
    km = new KeyedMessage('key', 'message'),
    payloads = [
        { topic: 'BertTopic', messages: 'first test message', partition: 0 },
        { topic: 'BertTopic', messages: 'second test message', partition: 0 }
    ];
producer.on('ready', function () {
    producer.send(payloads, function (err, data) {
        console.log(data);
        process.exit(0);
    });
});

producer.on('error', function (err) {
console.log('ERROR: ' + err.toString());
});

Which returned:

{ BertTopic: { '0': 0 } }

And I ran this second NodeJs program to consume the (last) messages:

var options = {
    fromOffset: 'latest'
};

var kafka = require('kafka-node'),
    Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(
        client,
        [
            { topic: 'BertTopic', partition: 0 }
        ],
        [
                {
                        autoCommit: false
                },
                options =
                {
                        fromOffset: 'latest'
                }
        ]
    );

Which returned:

{ topic: 'BertTopic',
  value: 'first test message',
  offset: 0,
  partition: 0,
  highWaterOffset: 2,
  key: null }
{ topic: 'BertTopic',
  value: 'second test message',
  offset: 1,
  partition: 0,
  highWaterOffset: 2,
  key: null }

I also have third NodeJs program to show all historical messages in the topic listed at my blog post https://bertrandszoghy.wordpress.com/2017/06/27/nodejs-querying-messages-in-apache-kafka/

Hope this helps someone out.

kafka: cannot produce messages to kafka inside docker

1 Answers