Kafka topic not found when running strimzi on docker-desktop k8s cluster

Question

I try to run strimzi on docker-desktop k8s, following this document.

I set up all the things.

$ kubectl get all -n my-kafka-project                                           
NAME                                              READY   STATUS    RESTARTS   AGE
pod/my-cluster-entity-operator-7ddb6d5b88-q2xg8   3/3     Running   0          97m
pod/my-cluster-kafka-0                            1/1     Running   0          98m
pod/my-cluster-zookeeper-0                        1/1     Running   0          98m

NAME                                          TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)                      AGE
service/my-cluster-kafka-0                    NodePort    10.99.207.179    <none>        9094:31805/TCP               98m
service/my-cluster-kafka-bootstrap            ClusterIP   10.103.193.53    <none>        9091/TCP,9092/TCP,9093/TCP   98m
service/my-cluster-kafka-brokers              ClusterIP   None             <none>        9091/TCP,9092/TCP,9093/TCP   98m
service/my-cluster-kafka-external-bootstrap   NodePort    10.97.198.62     <none>        9094:31314/TCP               98m
service/my-cluster-zookeeper-client           ClusterIP   10.101.206.203   <none>        2181/TCP                     98m
service/my-cluster-zookeeper-nodes            ClusterIP   None             <none>        2181/TCP,2888/TCP,3888/TCP   98m

NAME                                         READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/my-cluster-entity-operator   1/1     1            1           97m

NAME                                                    DESIRED   CURRENT   READY   AGE
replicaset.apps/my-cluster-entity-operator-7ddb6d5b88   1         1         1       97m

NAME                                    READY   AGE
statefulset.apps/my-cluster-kafka       1/1     98m
statefulset.apps/my-cluster-zookeeper   1/1     98m

And get nodeport:

$ kubectl get service my-cluster-kafka-external-bootstrap -n my-kafka-project -o=jsonpath='{.spec.ports[0].nodePort}{"\n"}'
31314

While this document uses minikube, I can access docker-desktop node by localhost, so I try to access localhost:31314, but I can't produce message to topic. It seems I can successfully connect to broker, but can't to topic.

./kafka-console-producer.sh --broker-list localhost:31314 --topic my-topic 
>Test
[2021-05-15 15:41:13,681] ERROR Error when sending message to topic my-topic with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
org.apache.kafka.common.errors.TimeoutException: Topic my-topic not present in metadata after 60000 ms.

I've checked with kafkacat, and topic itself is created successfully.

kafkacat -b localhost:31314 -L
Metadata for all topics (from broker -1: localhost:31314/bootstrap):
 1 brokers:
  broker 0 at 192.168.65.4:31805 (controller)
 4 topics:
  topic "__strimzi_store_topic" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "my-topic" with 3 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 0, replicas: 0, isrs: 0
    partition 2, leader 0, replicas: 0, isrs: 0
  topic "__strimzi-topic-operator-kstreams-topic-store-changelog" with 1 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
  topic "__consumer_offsets" with 50 partitions:
    partition 0, leader 0, replicas: 0, isrs: 0
    partition 1, leader 0, replicas: 0, isrs: 0
    partition 2, leader 0, replicas: 0, isrs: 0
    partition 3, leader 0, replicas: 0, isrs: 0
    partition 4, leader 0, replicas: 0, isrs: 0
    partition 5, leader 0, replicas: 0, isrs: 0
    ...

I don't fully understand mechanism how the message is sent to topic, so which point should I check next?

You should check what is the address seen by the Kafka brokers and the operator which might not be localhost. You can do kubectl get kafka -o yaml and check what is the bootstrap address for the external listener in the .status section. You can also check what is the advertised address configured in the brokers -> they print that configuratin at startup, so you should see it on the begginning of the log. If this address is different from localhost, you can try if that addres works with the kafkacat command or not. That might tell us something. — Jakub
The bootstrapServers address of external listener is 192.168.65.4:31314. The ip address is same as docker-desktop node's address, and the port is same as my-cluster-kafka-external-bootstrap Service's NodePort. kafkacat for the address above doesn't work. % ERROR: Failed to acquire metadata: Local: Broker transport failure — lipsum
So, it looks that the address which Docker reports inside the Kubernetes cluster (192.168.65.4) is not accessible from outside the cluster. If localhost works, you have probably two options. One, you can use the advertisedHost field to override the advertised addresses (strimzi.io/docs/operators/latest/full/…). — Jakub
Two, you can check if the localhost is advertised somewhere in the node (kubectl get node -o yaml) and use the preferredNodePortAddressType field to configure what will be used as the address (strimzi.io/docs/operators/latest/full/…). — Jakub

lipsum lipsum · Accepted Answer · 2021-05-15T16:17:09

Based on Jakub's comment, I can produce and consume message from host by changing Kafka resource in my-kafka-project namespace.

I've overridden the advertised listeners in the external listener of .spec.kafka.listeners.

from:

- name: external
  port: 9094
  tls: false
  type: nodeport

to:

- name: external
  port: 9094
  tls: false
  type: nodeport
  configuration:
    brokers:
    - broker: 0
      advertisedHost: localhost
      advertisedPort: 31314

Kafka topic not found when running strimzi on docker-desktop k8s cluster

1 Answers