GCP Health Check Failing with Istio Ingress Gateway Pod

Question

I'm trying to setup a GCE Ingress where it's using TLS from client to HTTPS load balancer back to the Istio Ingress Gateway but the health checks are failing for the HTTPS Load Balancer in GCE. In summary:

Client connection over TLS to HTTPS LB ---> HTTPS LB over TLS to Istio Ingress Gateway Pod

If I curl the healthcheck directly to the Istio Ingress pod over HTTP it works fine but over HTTPS it fails. Has anyone had luck with a similar setup?

HTTP

 curl -X GET http://10.40.0.135/ -H "user-agent: GoogleHC" -I -v
*   Trying 10.40.0.135...
* TCP_NODELAY set
* Connected to 10.40.0.135 (10.40.0.135) port 80 (#0)
> GET / HTTP/1.1
> Host: 10.40.0.135
> Accept: */*
> user-agent: GoogleHC
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< date: Fri, 13 Mar 2020 15:43:14 GMT
date: Fri, 13 Mar 2020 15:43:14 GMT
< content-length: 0
content-length: 0
< x-envoy-upstream-service-time: 0
x-envoy-upstream-service-time: 0
< server: istio-envoy
server: istio-envoy

HTTPS

 curl -X GET https://10.40.0.135/ -H "user-agent: GoogleHC" -I -v -k
*   Trying 10.40.0.135...
* TCP_NODELAY set
* Connected to 10.40.0.135 (10.40.0.135) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=nginx.example.com; O=SOME ORG
*  start date: Mar 12 20:23:01 2020 GMT
*  expire date: Mar 12 20:23:01 2021 GMT
*  issuer: O=example Inc.; CN=example.com
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55f864c6eea0)
> GET / HTTP/1.1
> Host: 10.40.0.135
> Accept: */*
> user-agent: GoogleHC
>
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 404
HTTP/2 404
< date: Fri, 13 Mar 2020 15:43:22 GMT
date: Fri, 13 Mar 2020 15:43:22 GMT
< server: istio-envoy
server: istio-envoy

<
* Curl_http_done: called premature == 0
* Connection #0 to host 10.40.0.135 left intact

Here's my configuration:

---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: istio-ingress
  namespace: istio-system
spec:
  backend:
      serviceName: istio-ingressgateway
      servicePort: 443
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: health-istio-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: healthcheck-https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
  - hosts:
    - '*'
    port:
      name: healthcheck-http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  annotations:
  name: istio-health
  namespace: istio-system
spec:
  gateways:
  - health-istio-gateway
  hosts:
  - '*'
  http:
  - match:
    - headers:
        user-agent:
          prefix: GoogleHC
      method:
        exact: GET
      uri:
          exact: /
    rewrite:
      authority: istio-ingressgateway:15020
      uri: /healthz/ready
    route:
    - destination:
        host: istio-ingressgateway
        port:
          number: 15020
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  labels:
    operator.istio.io/component: IngressGateway
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.4.3
    release: istio
  name: ingressgateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      name: http
      number: 80
      protocol: HTTP
    hosts:
    - '*'
  - port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - '*'

Answer 1

I found a similar issue on serverfault.
The issue on this link seems to come from the default path of a self-signed root CA.
That may be related to your issue.