Istio ingress not working with headless service

Question

I have deployed Kafka as a statefulset with zookeeper configured as leader selector, a headless service. Kafka is running absolutely fine as expected. However I am facing issues while configuring Istio to access kafka.

    $ kubectl get pods -owide | grep -i kafka
     kafka-mon-0   1/1     Running   0        3d1h    <IP>  

    $ kubectl get svc -owide | grep -i kafka
     kafka-mon-http      LoadBalancer   <IP>  <Ext-IP>    8080:30875/TCP  app=kafka-mon
     kafka-mon-svc       ClusterIP      None  <none>      8080/TCP    app=kafka-mon

If I configure Istio with Kakfa LoadBalancer Service, I am able to access the UI. However, if I use a headless service, then The UI itself is not accessible. I have tested with different other services as well, same is the case.

$ kubectl get gateway,virtualservice  | grep -i kafka

gateway.networking.istio.io/kafka-mon-gateway   4h
virtualservice.networking.istio.io/kafka-mon-vservice   4h

Istio works perfectly if Virtualservice configured with Load Balancer service, but not with the headless service. Please help me figure out the issue.

For Istio, I have deployed a Gateway router as internal-ingressgateway with http port- 80, https port-443 & A virtualservice with routing destination host as the Kafka-headless-service, It doesnt work, but it works if routing destination host is configured as Load Balancer service.

I am not able to troubleshoot the issue. Please suggest.

Have you added a selector on your headless service to point to your backend pods, check kubectl get endpoints for your service to check if the necessary route is created from service to backends — fatcook
Also, take a look at this github issue (github.com/istio/istio/issues/5005), it seems it is still not fixed. — mjkool
Hi @fatcook, I had verified that already. Headless service is pointing to the exact pods, it should refer to. the endpoints are same as the pod ips. My issue is, I am not getting any exception to verify for this. — Himansu

yog raj yog raj · Accepted Answer · 2019-11-15T06:43:02

I had this issue and I fixed it by adding a Service Entry. When we use a headless svc, istio is not sure where to direct the traffic to. U can add something similiar to below.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: kafka-se
  namespace: <If any namespace>
spec:
  hosts:
  - kafka.default.svc.cluster.local
  location: MESH_INTERNAL
  ports:
  - name: grpc
    number: 5445
    protocol: TCP
  resolution: DNS

Istio ingress not working with headless service

1 Answers