Expose individual Kafka brokers on Kubernetes through an ELB on AWS

Question

I am running a KOPS Kubernetes cluster on AWS, trying to make kubernetes-kafka example work with an Elastic Load Balancer. Here is the external services portion for two of the brokers:

kind: Service
apiVersion: v1
metadata:
  name: outside-0
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "0"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32400
    nodePort: 32400
  type: NodePort
---
kind: Service
apiVersion: v1
metadata:
  name: outside-1
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "1"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32401
    nodePort: 32401
  type: NodePort

Here is my attempt to expose those brokers via an ELB (actual FQDN replaced with my.copmany.com).

apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  annotations:
    dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32400
    name: outside
    targetPort: 32400
  selector:
    app: outside-0
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1
  annotations:
    dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32401
    name: outside
    targetPort: 32401
  selector:
    app: outside-1

Looking at AWS ELB console shows 0 of 3 instances available for each of the Kafka ELB brokers and producing to kafka-1.kafka.my.company.com:9094 using Kafka command line client times out. How can outside-0 NodePort service be exposed via kafka-0 LoadBalancer service? Or are there other approached to be considered?

@DmitryMinkovsky - thank you for the link, both of the top answers there have some good elements to them, first one exposes it through ec2-host:port and the second one uses elb:port. My challenge is each kafka broker needs to have it's own "permanent" URL like subdomain-on-company-domain:port, e.g. kafka-1.kafka.my.company.com:9094. — SergeyB
Yeah I've never quite figured it out myself. Fortunately in my situation I ended up realizing I could send data out from inside instead of opening Kafka up to the outside. If you figure this out, please post an answer. — Dmitry Minkovsky
The ELB DNS is permanent for the service. It will not change if you redeploy or even remove the deployment. It will be released if you delete the service. If you own your own DNS such as my.kafka.com you can alias it to the ELB DNS — Lev Kuznetsov

mbartoszek mbartoszek · Accepted Answer · 2018-02-08T16:38:36

Kakfa is very particular about clients needing direct access to the server who is the leader of a topic. To achieve this, I had done the following:

1) Setup the configmap to dynamically override values for the advertised.listeners based on the pod's ordinal value

POD_ID=${HOSTNAME##*-}

kafka-server-start.sh server.properties \
      --override advertised.listeners=INSIDE://`hostname -f`:9092,OUTSIDE://kafka-${POD_ID}.kafka.my.company.com:9094 \
      --override broker.id=${POD_ID} \
      --override listeners=INSIDE://:9092,OUTSIDE://:9094

2) Create a LoadBalancer service for each Kafka pod. Change the selector to match your kafka-pod-id.

    apiVersion: v1
    kind: Service
    metadata:
      name: kafka-0
      annotations:
        dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
    spec:
      externalTrafficPolicy: Local
      type: LoadBalancer
      ports:
      - port: 9094
        name: outside
        targetPort: 9094
      selector:
        app: kafka
        kafka-pod-id: "0"
---     
apiVersion: v1
        kind: Service
        metadata:
          name: kafka-1
          annotations:
            dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
        spec:
          externalTrafficPolicy: Local
          type: LoadBalancer
          ports:
          - port: 9094
            name: outside
            targetPort: 9094
          selector:
            app: kafka
            kafka-pod-id: "1"

Expose individual Kafka brokers on Kubernetes through an ELB on AWS

4 Answers