I'm new to dockers.
I have to create a kafka container which will have a consumer container whose job is to consume and save to mongodb.
my docker-compose looks like this:
version: '2'
services:
consumer:
image: springio/kafka-consumer
depends_on:
- kafka
- mongodb
zookeeper:
image: wurstmeister/zookeeper
ports:
- "2181:2181"
kafka:
image: wurstmeister/kafka
ports:
- "9092:9092"
environment:
KAFKA_ADVERTISED_HOST_NAME: kafka
KAFKA_CREATE_TOPICS: "test:1:1"
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
volumes:
- /var/run/docker.sock:/var/run/docker.sock
mongodb:
image: mongo
container_name: "mongodb"
environment:
- MONGO_DATA_DIR=/data/db
- MONGO_LOG_DIR=/dev/null
volumes:
- ./data/db:/data/db
ports:
- 27017:27017
As far as I understand I couldn't use KAFKA_ADVERTISED_HOST_NAME as 'localhost' because then the consumer container would not be able to reach it (since it would try to connect to its own container localhost). Therefore I name it 'kafka'.
Now, I want to run a producer (not a container) on my host machine (same machine the containers run).
The problem is, the minute I changed the KAFKA_ADVERTISED_HOST_NAME to be different than localhost, I cannot produce to kafka from the host machine producer.
Why can't I produce to kafka? I thought when I stated ports: 9092:9092, I actually exported that port to the host machine, am I wrong?
What is the best way/s to reach the kafka from the host machine without using an additional container for the producer?
Any help would be appreciated!!
EDIT:
When I try to produce a message using kafka-console-producer, only when the KAFKA_ADVERTISED_HOST_NAME is not 'localhost', I get the following message:
ERROR Error when sending message to topic test with key: null, value: 3 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for test-0: 1503 ms has passed since batch creation plus linger time
9092:9092
mapping will allow you to reach that container using 'localhost` from outside of the container. Regardless of advertised listeners property, I believe – OneCricketeerkafka:9092
external to the container or its network will never work. You need to port map, as you have, then use localhost:9092... The same goes for Mongo – OneCricketeer