Istio Replicated Control Planes - 503 unavailable

Question

While trying to configure Repilicated Control Planes as described in this guide: https://istio.io/latest/docs/setup/install/multicluster/gateways/

After doing all the configuration the "sleep" application is unable to communicate with "httpbin" application as described in the documentation. The result of the test is always the same about 503 Service Unavailable error:

kubectl exec --context=kontiki $SLEEP_POD -n multi-test -c sleep -- curl -k -I httpbin.multi-test-bar.global:8000/headers
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0    91    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 503 Service Unavailable
content-length: 91
content-type: text/plain
date: Wed, 07 Oct 2020 13:55:19 GMT
server: envoy

The relevant logs found are:

istio-proxy container in sleep pod on origin cluster

[2020-10-07T13:58:21.775Z] "HEAD /headers HTTP/1.1" 503 UF,URX "-" "-" 0 0 137 - "-" "curl/7.69.1" "5ccf05b7-d0e3-9e38-a581-8c0bdabc98b3" "httpbin.multi-t0" "10.14.10.99:31383" outbound|8000||httpbin.multi-test-bar.global - 240.0.0.2:8000 172.17.141.20:59704 - default

ingress pods on destination cluster

[2020-10-07T13:58:21.900Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.184.60:15443 172.31.4.248:62395 - -
[2020-10-07T13:58:21.814Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.133.59:15443 172.31.4.209:38326 - -

Istio 1.7.3 is deployed with Istio-operator on vanilla k8s clusters with version 1.17. Certificates are configured as described in the referenced guide, ServiceEntry created for httpbin is the following:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: httpbin-multi-test-bar
spec:
  hosts:
  - httpbin.multi-test-bar.global
  location: MESH_INTERNAL
  ports:
  - name: http1
    number: 8000
    protocol: http
  resolution: DNS
  addresses:
  - 240.0.0.2
  endpoints:
  - address: cluster-2
    network: external
    ports:
      http1: 31383 # nodePort tls exposed port via our proxy
  - address: istio-egressgateway.istio-system.svc.cluster.local
    ports:
      http1: 15443

This error is also referenced in this #Issue where for other people, this very same thing was happening and the way they fixed it was to moving to a previous Istio version like 1.6.8. edit: I can confirm that my configuration works with version 1.6.8, but it fails with 1.7.3.

Can you please help me understand what's happening or how could it be fixed?

Jakub Jakub · Accepted Answer · 2020-10-08T12:53:26

There are more issues reported with that specific issue:

and there is no answer so far about how to fix it.

I would suggest to wait for the answer from istio devs in the issue you mentioned and use 1.6.8 untill that will be solved.

The issue itself might be related to dns changes in 1.8, but these are just my thoughts.

Starting with Istio 1.8, the Istio agent on the sidecar will ship with a caching DNS proxy, programmed dynamically by Istiod.

There are more informations about dns changes:

And there are preliminary docs for 1.8 multi cluster installation.

Istio Replicated Control Planes - 503 unavailable

1 Answers