Kafka on kubernetes cluster with Istio

Question

I have k8s cluster with Istio v1.6.4. The sidecar injection is disabled by default. I have Kafka cluster running on this k8s installed with strimzi kafka operator. The Kafka cluster works without any problems when kafka as well as client pods doesn't have Istio-proxy injected. My problem: When I create a pod with kafka client and Istio-proxy injected I can't connect to Kafka cluster. The logs on client side:

java.io.IOException: Connection reset by peer

and on the server side: org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 369295616 larger than 104857600)

After some googling and checking the Istio-proxy logs it turns out the problem is that Istio-proxy connects to kafka plaintext endpoint with TLS. I can workaround this by setting the default PeerAuthentication with mtls.mode: DISABLED but I don't want to set global setting for it.

What is strange if I create a simple k8s service and run the netcat "server" on pod running kafka server and netcat "client" on pod running kafka client - everything works fine.

I have 2 question:

Why the kafka Istio-proxy behaves different when connecting to Kafka cluster than other TCP connections (like using nc)?
How to disable mtls for one host only? I was playing with PeerAuthentication but no luck...

As far as I understand what you need to do is to add destination rule with trafficPolicy: tls: mode: SIMPLE for this one host. Take a look at documentation here. let me know if this solves the problem. — Jakub
Thank you for your help! Unfortunately, it didn't help. I tried mode SIMPLE as well as DISABLE. The only option which works for me is global PeerAuthentication with spec.mtls.mode: DISABLE. — gkocur
1.Everything you deploy is in default namespace? 2.Can you show the client service and destination rule yaml? 3.What about peer authentication for all workloads under some namespace as mentioned in documentation instead of doing it globally? 4.Is Kafka Service exposed as a headless service? — Jakub
Question 4. pointed me to the solution. Strimzi kafka creates 2 services: kafka-bootstrap which is a regular service (with IP) and kafka-brokers which is headless service. I was using the bootstrap and switching to brokers (together with DestinationRule) solved this issue. Thank you! Will write a detailed answer. — gkocur

gkocur gkocur · Accepted Answer · 2020-07-15T11:56:32

With jt97's help I was able to solve this problem.

As I wrote I'm using Strimzi Operator to install kafka cluster on k8s. It creates 2 services:

kafka-bootstrap - which is a regular service with ClusterIP
kafka-brokers - a headless service.

In my case the full name of the services are kafka-kafka-operated-kafka-bootstrap and kafka-kafka-operated-kafka-brokers, respectively.

I created a DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: kafka-no-tls
spec:
  host: "kafka-kafka-operated-kafka-brokers"
  trafficPolicy:
    tls:
      mode: DISABLE

and used the headless service when connecting to kafka:

kafka-topics --bootstrap-server kafka-kafka-operated-kafka-brokers:9092 --list
__consumer_offsets
_schemas

and it worked as expected.

BTW, setting tls.mode to SIMPLE didn't help.

To be honest I still don't understand why in this particular case Istio-proxy by default (without DestinationRule) tries to connect with TLS - according to documentation:

By default, Istio tracks the server workloads migrated to Istio proxies, and configures client proxies to send mutual TLS traffic to those workloads automatically, and to send plain text traffic to workloads without sidecars.

Kafka on kubernetes cluster with Istio

1 Answers