Can't connect spring application to debezium kafka

Question

I'm trying to connect my spring application to the debezium/kafka:0.10 but no success. Originally the app uses wurstmeister/kafka image but now I'm introducing a new functionality with debezium and want to use only one kafka image for the both. I can't connect neither app to debezium kafka or debezium to wurstmeister kafka. Do someone knows is it possible to use only one image for both? Any ideas are welcome :)

The problem is that the app is not able to create the topic's into debezium container. And also debezium is not able to work with wurstmeister kafka image.

ERROR 18864 --- [           main] o.springframework.kafka.core.KafkaAdmin  : Could not configure topics

org.springframework.kafka.KafkaException: Timed out waiting to get existing topics; nested exception is java.util.concurrent.TimeoutException
    at org.springframework.kafka.core.KafkaAdmin.lambda$checkPartitions$2(KafkaAdmin.java:235) ~[spring-kafka-2.2.3.RELEASE.jar:2.2.3.RELEASE]
    at java.base/java.util.HashMap.forEach(HashMap.java:1336) ~[na:na]

services:
  zookeeper:
    image: debezium/zookeeper:0.10
    ports:
      - 2181:2181
      - 2888:2888
      - 3888:3888
  kafka:
    image: debezium/kafka:0.10
    ports:
      - 9092:9092
    links:
      - zookeeper
    environment:
      - ZOOKEEPER_CONNECT=zookeeper:2181
      - ADVERTISED_HOST_NAME=10.0.75.1
  connect:
    image: debezium/connect:0.10
    ports:
      - 8083:8083
    links:
      - kafka
      - mysql
    environment:
      - BOOTSTRAP_SERVERS=kafka:9092
      - GROUP_ID=1
      - CONFIG_STORAGE_TOPIC=my_connect_configs
      - OFFSET_STORAGE_TOPIC=my_connect_offsets
      - STATUS_STORAGE_TOPIC=my_connect_statuses

Possible duplicate of Kafka Cannot Configure Topics on Application Startup, but Later Can Communicate — Naros
Then might I suggest you provide more details about your environment, code, and configuration? You've tried two different images with the same outcome. It's hard to diagnose the underlying problem without more details. — Naros
Sounds like you misconfigured ADVERTISED_HOST_NAME property. Debezium works with any Kafka container — OneCricketeer
ADVERTISED_HOST_NAME -> 172.0.0.1:9092 if someone has a example configuration with debezium working with different kafka than debezium/kafka is welcome — sherybedrock

Anand Immannavar Anand Immannavar · Accepted Answer · 2020-05-14T05:15:52

Along with ADVERTISED_HOST_NAME, You need to add ADVERTISED_LISTENERS to container environment.

ADVERTISED_LISTENERS - Broker will register this value in zookeeper and when the external world wants to connect to your Kafka Cluster they can connect over the network which you provide in ADVERTISED_LISTENERS property.

example:

environment:
    - ADVERTISED_HOST_NAME=<Host IP>
    - ADVERTISED_LISTENERS=PLAINTEXT://<Host IP>:9092

Can't connect spring application to debezium kafka

1 Answers