Need help troubleshooting Istio IngressGateway HTTP ERROR 503

Question

My Test Environment Cluster has the following configurations :

Global Mesh Policy (Installed as part of cluster setup by our org) : output of kubectl describe MeshPolicy default

Name:         default
Namespace:
Labels:       operator.istio.io/component=Pilot
              operator.istio.io/managed=Reconcile
              operator.istio.io/version=1.5.6
              release=istio
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"authentication.istio.io/v1alpha1","kind":"MeshPolicy","metadata":{"annotations":{},"labels":{"operator.istio.io/component":...
API Version:  authentication.istio.io/v1alpha1
Kind:         MeshPolicy
Metadata:
  Creation Timestamp:  2020-07-23T17:41:55Z
  Generation:          1
  Resource Version:    1088966
  Self Link:           /apis/authentication.istio.io/v1alpha1/meshpolicies/default
  UID:                 d3a416fa-8733-4d12-9d97-b0bb4383c479
Spec:
  Peers:
    Mtls:
Events:  <none>

The above configuration I believe enables services to receive connections in mTls mode.

DestinationRule : Output of kubectl describe DestinationRule commerce-mesh-port -n istio-system

Name:         commerce-mesh-port
Namespace:    istio-system
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"networking.istio.io/v1alpha3","kind":"DestinationRule","metadata":{"annotations":{},"name":"commerce-mesh-port","namespace"...
API Version:  networking.istio.io/v1beta1
Kind:         DestinationRule
Metadata:
  Creation Timestamp:  2020-07-23T17:41:59Z
  Generation:          1
  Resource Version:    33879
  Self Link:           /apis/networking.istio.io/v1beta1/namespaces/istio-system/destinationrules/commerce-mesh-port
  UID:                 4ef0d49a-88d9-4b40-bb62-7879c500240a
Spec:
  Host:  *
  Ports:
    Name:      commerce-mesh-port
    Number:    16443
    Protocol:  TLS
  Traffic Policy:
    Tls:
      Mode:  ISTIO_MUTUAL
Events:      <none>

Istio Ingress-Gateway :

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: finrpt-gateway
  namespace: finrpt
spec:
  selector:
    istio: ingressgateway # use Istio's default ingress gateway
  servers:
  - port:
      name: https
      number: 443
      protocol: https
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"
  - port:
      name: http
      number: 80
      protocol: http
    tls:
      httpsRedirect: true
    hosts:
    - "*"

I created a secret to be used for TLS and using that to terminate the TLS traffic at the gateway (as configured in mode SIMPLE)

Next, I configured my VirtualService in the same namespace and did a URL match for HTTP :

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: finrpt-virtualservice
  namespace: finrpt
spec:
  hosts:
  - "*"
  gateways:
  - finrpt-gateway
  http:
  - match:
    - queryParams:
        target:
          exact: "commercialprocessor"
      ignoreUriCase: true
    route:
    - destination:
        host: finrpt-commercialprocessor
        port:
          number: 8118

The Service CommercialProcessor (ClusterIP) is expecting traffic on HTTP/8118.

With the above setting in place, when I browse to the External IP of my Ingress-Gateway, first I get a certificate error (expected as I am using self-signed for testing) and then on proceeding I get HTTP Error 503.

I am not able to find any useful logs in the gateway, I am wondering if the gateway is unable to communicate to my VirtualService in plaintext (TLS termination) and it is expecting https but I have put it as http? Any help is highly appreciated, I am very new to Istio and I think I might be missing something naive here.

My expectation is : I should be able to hit the Gateway with https, gateway does the termination and forwards the unencrypted traffic to the destination configured in the VirtualService on HTTP port based on URL regex match ONLY (I have to keep URL match part constant here).

Gateway looks fine, If I understand documentation properly then ignoreUriCase: true won´t work with queryParamas, only with uri. I dont know what is that commerce-mesh-port destination rule for? Could you try to add destination rule with trafficPolicy: tls: mode: SIMPLE for your virtual service as mentioned in documentation? — Jakub
Thanks for the reply @jt97. I did try to create a new destination rule with mode SIMPLE for all hosts in the default namespace but did not work. I even tried DISABLE, but it did not work either. I think the commerce-mesh-port has to do something with contacting the kube api server as it seems likely from the port number 16443, but not sure. Lastly, I deleted any ServiceEntry/DestinationRule related to ISTIO_MUTUAL and tried again but no luck yet :( Still getting the 503 error. — Jim
You mentioned ServiceEntry, you mean virtual service? Or do you use service entry and your app is not in the mesh? Do you have correct name in your service as mentioned here? Could you try to first deploy destination rule and then deploy virtual service? Did you label the namespace with istio-injection=enabled? Could you try to install bookinfo app and check if it works? — Jakub
I see the answer posted by @Jakub helped you, please consider upvoting/accepting it so other users can see it was useful. Please refer to What should I do when someone answers my question? — Mark Watney

Jakub Jakub · Accepted Answer · 2020-08-05T09:00:32

As 503 often occurs and it´s hard to find the issue I set up little troubleshooting answer, there are another questions with 503 error which I encountered for several months with answers, useful informations from istio documentation and things I would check.

Examples with 503 error:

Common cause of 503 errors from istio documentation:

Few things I would check first:

Check services ports name, Istio can route correctly the traffic if it knows the protocol. It should be <protocol>[-<suffix>] as mentioned in istio documentation.
Check mTLS, if there are any problems caused by mTLS, usually those problems would result in error 503.
Check if istio works, I would recommend to apply bookinfo application example and check if it works as expected.
Check if your namespace is injected with kubectl get namespace -L istio-injection
If the VirtualService using the subsets arrives before the DestinationRule where the subsets are defined, the Envoy configuration generated by Pilot would refer to non-existent upstream pools. This results in HTTP 503 errors until all configuration objects are available to Pilot.

Hope you find this useful.

Need help troubleshooting Istio IngressGateway HTTP ERROR 503

1 Answers