How to access kafka installed outside kubernetes cluster from a service provisioned inside a kubernetes cluster

Question

My setup is like, I have a producer service provisioned as part of minikube cluster and it is trying to publish messages to a kafka instance running on the host machine.

I have written a kafka service and endpoints yaml as follows:

kind: Service
apiVersion: v1
metadata:
  name: kafka
spec:
  ports:
    - name: "broker"
      protocol: "TCP"
      port: 9092
      targetPort:  9092
      nodePort: 0
---
kind: Endpoints
apiVersion: v1
metadata:
  name: kafka
  namespace: default
subsets:
  - addresses:
      - ip: 10.0.2.2
    ports:
      - name: "broker"
        port: 9092

The ip address of the host machine from inside the minikube cluster mentioned in the endpoint is acquired from the following command:

minikube ssh "route -n | grep ^0.0.0.0 | awk '{ print \$2 }'"

The problem I am facing is that the topic is getting created when producer tries to publish message for the first time but no messages are getting written on to that topic.

Digging into the pod logs, I found that producer is trying to connect to kafka instance on localhost or something (not really sure of it).

2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] - 
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer] 
Connection to node 0 (omkara/127.0.1.1:9092) could not be established. Broker may not be available.

Following which I suspected that probably I need to modify server.properties with the following change:

listeners=PLAINTEXT://localhost:9092

This however resulted in the change in the ip address in the log:

2020-05-17T19:09:43.021Z [warn] org.apache.kafka.clients.NetworkClient [] - 
[Producer clientId=/system/sharding/kafkaProducer-greetings/singleton/singleton/producer] 
Connection to node 0 (omkara/127.0.0.1:9092) could not be established. Broker may not be available.

I am not sure what ip address must be mentioned here? Or what is an alternate solution? And if it is even possible to connect from inside the kubernetes cluster to the kafka instance installed outside the kubernetes cluster.

you need to set listeners=PLAINTEXT://localhost:9092 exactly yo IP you specify in Endpoints config, i.e. listeners=PLAINTEXT://10.0.2.2:9092 — Anton Matsiuk
@AntonMatsiuk org.apache.kafka.common.KafkaException: Socket server failed to bind to 10.0.2.2:9092: Cannot assign requested address. — iamsmkr
@GiorgosMyrianthous I have already tried that. But which address do you propose anyway? — iamsmkr

iamsmkr iamsmkr · Accepted Answer · 2020-05-20T12:00:55

Since producer kafka client is on the very same network as the brokers, we need to configure an additional listener like so:

listeners=INTERNAL://0.0.0.0:9093,EXTERNAL://0.0.0.0:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
advertised.listeners=INTERNAL://localhost:9093,EXTERNAL://10.0.2.2:9092
inter.broker.listener.name=INTERNAL

We can verify messages in topic like so:

kafka-console-consumer.sh --bootstrap-server INTERNAL://0.0.0.0:9093 --topic greetings --from-beginning
{"name":"Alice","message":"Namastey"}

You can find a detailed explaination on understanding & provisioning kafka listeners here.

How to access kafka installed outside kubernetes cluster from a service provisioned inside a kubernetes cluster

1 Answers