IstIO egress gateway gives HTTP 503 error

Question

I have the following manifest for deploying a IstIO egress gateway routing:

---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: REDACTED-egress-se
spec:
  hosts:
  - sahfpxa.REDACTED
  ports:
  - number: 8080
    name: http-port
    protocol: HTTP
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: sahfpxa-REDACTED-egress-gw
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 8080
      name: http
      protocol: HTTP
    hosts:
    - sahfpxa.REDACTED
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-sahfpxa-REDACTED
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: sahfpxa
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-sahfpxa-REDACTED-through-egress-gateway
spec:
  hosts:
  - sahfpxa.REDACTED
  gateways:
  - REDACTED/REDACTED-egress-gw
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 8080
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: sahfpxa
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - REDACTED/sahfpxa-REDACTED-egress-gw
      port: 8080
    route:
    - destination:
        host: sahfpxa.REDACTED
        port:
          number: 8080
      weight: 100

But I get a connection refused from the sidecar istio-proxy container Pod of the affected namespace and a HTTP 503 error from the workload container in that namespace.

Any ideas what could be wrong with the configuration or how I can debug it?

Thanks in advance.

Best regards, rforberger

Hi, 1. Can You be more specific with how You get the connection refused errors? What specific command did You use and from where? 2. Is the injected deployment pod accessible from within cluster/namespace? 3. What istio version do You have? — Piotr Malec
Hi @PiotrMalec 1. I get the connection refused from the envoy sidecar of my workload container from which I want to reach the external service sahfpxa.REDACTED through the egressgateway. 2. You mean if I can reach the injected deployment pod from another pod ? 3. Istio 1.4.0 (just upgraded, but the issue persists) — Ronny Forberger
Sorry I got a little bit confused about source and destinations services for this issue. So checking if the service is accessible withing cluster makes no sense. Instead check If the external service can be reached from cluster node. Have You tried using curl with --verbose option? Its HTTP protocol so there could be some useful information. — Piotr Malec
Hi @PiotrMalec The external service can be reached from the cluster node, also from the egressgateway pod. Curl --verbose shows the following: * Trying 10.224.19.37:8080... * TCP_NODELAY set * Connected to sahfpxa.REDACTED (10.224.19.37) port 8080 (#0) > POST /REDACTED HTTP/1.1 > Host: sahfpxa.REDACTED:8080 > User-Agent: curl/7.66.0 > Accept: */* > * Mark bundle as not supporting multiuse < HTTP/1.1 503 Service Unavailable < content-length: 91 < content-type: text/plain < date: Wed, 04 Dec 2019 17:35:29 GMT < server: envoy < — Ronny Forberger
Try to add location: MESH_EXTERNAL under spec for ServiceEntry named REDACTED-egress-se. — Piotr Malec

Piotr Malec Piotr Malec · Accepted Answer · 2019-12-09T19:29:01

There were few errors in Your deployment manifest like DestinationRule was not pointing at Your ServiceEntry.

You can try to match Yours with these manifest files:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: etth
spec:
  hosts:
  - etth.pl
  ports:
  - number: 8080
    name: http-port
    protocol: HTTP
  resolution: DNS

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - etth.pl
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-cnn
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: etth

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-cnn-through-egress-gateway
spec:
  hosts:
  - etth.pl
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: etth
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 80
    route:
    - destination:
        host: etth.pl
        port:
          number: 8080
      weight: 100

You can check if routes are present in: istioctl pc routes $(kubectl get pods -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}' -n istio-system).istio-system -o json

$ istioctl pc routes $(kubectl get pods -l istio=egressgateway -o jsonpath='{.items[0].metadata.name}' -n istio-system).istio-system -o json
[
    {
        "name": "http.80",
        "virtualHosts": [
            {
                "name": "etth.pl:80",
                "domains": [
                    "etth.pl",
                    "etth.pl:80"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/",
                            "caseSensitive": true
                        },
                        "route": {
                            "cluster": "outbound|8080||etth.pl",
                            "timeout": "0s",
                            "retryPolicy": {
                                "retryOn": "connect-failure,refused-stream,unavailable,cancelled,resource-exhausted,retriable-status-codes",
                                "numRetries": 2,
                                "retryHostPredicate": [
                                    {
                                        "name": "envoy.retry_host_predicates.previous_hosts"
                                    }
                                ],
                                "hostSelectionRetryMaxAttempts": "5",
                                "retriableStatusCodes": [
                                    503
                                ]
                            },
                            "maxGrpcTimeout": "0s"
                        },
                        "metadata": {
                            "filterMetadata": {
                                "istio": {
                                    "config": "/apis/networking/v1alpha3/namespaces/default/virtual-service/direct-cnn-through-egress-gateway"
                                }
                            }
                        },
                        "decorator": {
                            "operation": "etth.pl:8080/*"
                        },
                        "typedPerFilterConfig": {
                            "mixer": {
                                "@type": "type.googleapis.com/istio.mixer.v1.config.client.ServiceConfig",
                                "disableCheckCalls": true,
                                "mixerAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "etth.pl"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "etth.pl"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "default"
                                        }
                                    }
                                },
                                "forwardAttributes": {
                                    "attributes": {
                                        "destination.service.host": {
                                            "stringValue": "etth.pl"
                                        },
                                        "destination.service.name": {
                                            "stringValue": "etth.pl"
                                        },
                                        "destination.service.namespace": {
                                            "stringValue": "default"
                                        }
                                    }
                                }
                            }
                        }
                    }
                ]
            }
        ],
        "validateClusters": false
    },
    {
        "virtualHosts": [
            {
                "name": "backend",
                "domains": [
                    "*"
                ],
                "routes": [
                    {
                        "match": {
                            "prefix": "/stats/prometheus"
                        },
                        "route": {
                            "cluster": "prometheus_stats"
                        }
                    }
                ]
            }
        ]
    }
]

IstIO egress gateway gives HTTP 503 error

1 Answers