Kubernetes pods can not make https request after deploying istio service mesh

Question

I am exploring the istio service mesh on my k8s cluster hosted on EKS(Amazon).

I tried deploying istio-1.2.2 on a new k8s cluster with the demo.yml file used for bookapp demonstration and most of the use cases I understand properly.

Then, I deployed istio using helm default profile(recommended for production) on my existing dev cluster with 100s of microservices running and what I noticed is my services can can call http endpoints but not able to call external secure endpoints(https://www.google.com, etc.)

I am getting :

curl: (35) error:1400410B:SSL routines:CONNECT_CR_SRVR_HELLO:wrong version number

Though I am able to call external https endpoints from my testing cluster.

To verify, I check the egress policy and it is mode: ALLOW_ANY in both the clusters.

Now, I removed the the istio completely from my dev cluster and install the demo.yml to test but now this is also not working.

I try to relate my issue with this but didn't get any success.

https://discuss.istio.io/t/serviceentry-for-https-on-httpbin-org-resulting-in-connect-cr-srvr-hello-using-curl/2044

I don't understand what I am missing or what I am doing wrong.

Note: I am referring to this setup: https://istio.io/docs/setup/kubernetes/install/helm/

Do you have any Service, anywhere in your cluster, listening on port 443? — David Maze
If you are asking for k8s service then Yes I have lot of cluster IP type service in my cluster that are listening on 80 and 443. Close to 50 services. I am not able to relate how it will effect Istio working? cluster-autoscaler-metrics, kubernetes-dashboard and many more. Even kubernetes have kubernetes clusterIP type service 127.0.0.1 on 443. — Vaibhav Jain

David Maze David Maze · Accepted Answer · 2019-08-24T19:47:55

This is most likely a bug in Istio (see for example istio/istio#14520): if you have any Kubernetes Service object, anywhere in your cluster, that listens on port 443 but whose name starts with http (not https), it will break all outbound HTTPS connections.

The instance of this I've hit involves configuring an AWS load balancer to do TLS termination. The Kubernetes Service needs to expose port 443 to configure the load balancer, but it receives plain unencrypted HTTP.

apiVersion: v1
kind: Service
metadata:
  name: breaks-istio
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: arn:...
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
spec:
  selector: ...
  ports:
    - name: http-ssl # <<<< THIS NAME MATTERS
      port: 443
      targetPort: http

When I've experimented with this, changing that name: to either https or tcp-https seems to work. Those name prefixes are significant to Istio, but I haven't immediately found any functional difference between telling Istio the port is HTTPS (even though it doesn't actually serve TLS) vs. plain uninterpreted TCP.

You do need to search your cluster and find every Service that listens to port 443, and make sure the port name doesn't start with http-....

Kubernetes pods can not make https request after deploying istio service mesh

1 Answers