Istio 503:s between (Public) Gateway and Service

Question

I've been playing around with my Istio cluster configuration and I've ended up in a state I can't debug my way out of.

I have the SDS+Gateway with a public IP configured. I have deployed the Istio HelloWorld app on port 5000. I can:

exec into istio-proxy on the helloworld-[rnd] pod and curl localhost:5000/hello - this works fine
check with istioctl proxy-config cluster (and such) from https://istio.io/docs/ops/troubleshooting/network-issues/ and https://istio.io/docs/ops/troubleshooting/proxy-cmd/ — all report OK to everything, SYNC:ed and such
I can do kubectl exec istio-ingressgateway-[rnd] /bin/bash and then curl helloworld.mynamespace:5000/hello successfully (it returns Hello version: v2 ...

But I can't make the ingressgateway actually return anything but 503 when querying its publicly bound IP. If I query without the /hello path, it returns 404 instead, so it's obviously trying to route to the helloworld service/deployment and failing.

So I'm in the state where I can actually contact my helloworld service from the Istio Ingress Gateway, when asking the gateway itself curl localhost/hello -i, or from ourside the network curl -i http://35.x.y.z/hello I always get 503 Service Unavailable Back

I don't have any DestinationRule nor Policy applying to helloworld, and I have Istio in strict mTLS.

I could previously today access (other) services via the ingress gateway, but then I started cleaning things up (to the point when I only have the helloworld service VirtualService+Gateway and no others), and now it doesn't work. It should be possible to debug.

What is wrong?

Not related (that I can tell):

Kubernetes Istio ingress gateway responds with 503 always (I don't have clusterIP: None)
Accessing service using istio ingress gives 503 error when mTLS is enabled (after k exec -c istio-proxy helloworld-[rnd] -- curl http://localhost:15000/logging?level=true, the istio-proxy envoy doesn't receive any calls from istio-ingressgateway at all; the traffic never leaves the ingress pod, unlike this question)
I have CNI + GKE Network Policy enabled (but turning it off didn't help) and a Calico-allow-all rule didn't help, so it should not be this; also, I can curl from ingressgateway, so there's connectivity
https://github.com/istio/istio/tree/master/samples/helloworld — config

Piotr Malec Piotr Malec · Accepted Answer · 2019-10-24T10:38:16

First of all to use curl with SDS gateway You need to use it as described in Istio documentation.

$ curl -v -HHost:httpbin.example.com \
--resolve httpbin.example.com:$SECURE_INGRESS_PORT:$INGRESS_HOST \
--cacert httpbin.new.example.com/2_intermediate/certs/ca-chain.cert.pem \
https://httpbin.example.com:$SECURE_INGRESS_PORT/status/418

...
HTTP/2 418
...
-=[ teapot ]=-

   _...._
 .'  _ _ `.
| ."` ^ `". _,
\_;`"---"`|//
  |       ;/
  \_     _/
    `"""`

Secondly according to Istio documentation using strict mTLS (mutual TLS) authentication policy requires both services to be running TLS communication. In Your case You are trying to access plain text (HTTP) service with Istio that is using TLS. This causes mutual TLS configuration conflict.

You can verify that with istioctl command in this section of documentation:

The istioctl command provides an option for this purpose. You can do:
$ istioctl authn tls-check $CLIENT_POD httpbin.default.svc.cluster.local
HOST:PORT                                  STATUS     SERVER     CLIENT     AUTHN POLICY        DESTINATION RULE
httpbin.default.svc.cluster.local:8000     OK         mTLS       mTLS       default/            default/istio-system
Where $CLIENT_POD is the ID of one of the client service’s pods.

Refer to Verify mutual TLS configuration for more information.

To resolve this issue mTLS has to be turned off for this service so that Istio accepts connection from plain text to TLS services. Follow this guide to create destination rule that allows non TLS communication for specified service

To confirm that this is causing this issue You can temporarily enable Permissive mode.

Edit:

From the link you provided in the last deployment file helloworld.yaml there is no targetPort and this is why nginx is unreachable.

This is how it should look:

apiVersion: v1
kind: Service
metadata:
  name: helloworld
  labels:
    app: helloworld
spec:
  ports:
  - port: 5000
    name: http
    targetPort: 80
  selector:
    app: helloworld
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v1
  labels:
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v1
  template:
    metadata:
      labels:
        app: helloworld
        version: v1
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v1
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: helloworld-v2
  labels:
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: helloworld
      version: v2
  template:
    metadata:
      labels:
        app: helloworld
        version: v2
    spec:
      terminationGracePeriodSeconds: 0
      containers:
      - name: helloworld
        image: docker.io/istio/examples-helloworld-v2
        resources:
          requests:
            cpu: "100m"
        imagePullPolicy: IfNotPresent #Always
        ports:
        - containerPort: 5000

Istio 503:s between (Public) Gateway and Service

1 Answers