Read Docker Kafka messages populated by debezium postgres connect

0
votes

I am using the Debezium Postgres connector. I have two tables in Postgres named 'publications' and 'comments'. kafka and zookeeper are running in docker containers as per the standard examples. The postgres is running locally. After using the debezium postgres connect, I have the following topics :

$ bin/kafka-topics.sh --list --zookeeper localhost:2181
  • __consumer_offsets
  • dbserver1.public.comments
  • dbserver1.public.publications
  • my_connect_configs
  • my_connect_offsets
  • my_connect_statuses

I would like to see a list of messages in the topic:

$ bin/kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic dbserver1.public.publications

[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03 21:55:16,289] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03 21:55:16,443] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03 21:55:16,721] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03 21:55:17,145] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2019-06-03 21:55:18,017] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) ^CProcessed a total of 0 messages

$ bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic dbserver1.public.publications

[2019-06-03 21:55:16,180] WARN [Consumer clientId=consumer-1, groupId=console-consumer-5221] Connection to node -1 (kafka/23.202.231.166:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

How do I specify the correct value for bootstrap-server? Thanks.

2
postgresqldockerapache-kafkadebezium
Possible duplicate of Connect to Kafka running in Docker from local machine – OneCricketeer
Your host machine doesn't know what kafka:9092 is, only the Docker network does... Did you mean to use localhost like you did for Zookeeper? – OneCricketeer
Yes, I meant to use localhost, but if you see the output above, i tried with localhost:9092 and it has the same problem – Anand Swaminathan
Your advertised listeners are incorrectly configured, then. The Debezium docker examples assume you run those commands within the container. Please refer to the duplicate post I marked... You can use any Kafka container with Debezium, it doesn't have to be one they provide – OneCricketeer

2 Answers

0
votes

I am assuming you are trying to connect to kafka broker from an external server.

Since you have already mentioned your Kafka and Zookeeper instances are running from docker images. You need to identify your docker images external port corresponding to 9092 as well as its external IP address and you have to those along with --bootstrap-server parameter while executing command kafka-console-consumer.sh

0
votes

If you are running the kafka-console-consumer.sh outside of docker then you should use localhost hostname. If inside the Docker container then make sure it is in container that sees kafka hostname.