TLS handshake through Istio ingress gateway fails (tlsMode=passthrough)

Question

A TLS handshake from an external client to a server inside a Kubernetes cluster fails. This is about understanding why.

I've configured an Istio ingress gateway to pass through TLS received on port 15433, and route it to the server on port 433.

The ingress gateway logs shows activity when the client attempts the TLS handshake, but not the server logs, nor the istio-proxy logs.

TLS client:

openssl s_client \ 
        -connect [redacted]-[redacted].us-west-2.elb.amazonaws.com:15443 \ 
        -servername myservice.mynamespace \ 
        -CAfile /path/to/ca.cert \ 
        -cert /path/to/cert.pem \ 
        -key /path/to/cert.key <<< "Q"

logs

CONNECTED(00000006) 
140090868934296:error:140790E5:SSL routines:ssl23_write:ssl handshake failure:s23_lib.c:177: 
--- 
no peer certificate available 
--- 
No client certificate CA names sent 
--- 
SSL handshake has read 0 bytes and written 298 bytes 
--- 
New, (NONE), Cipher is (NONE) 
Secure Renegotiation IS NOT supported 
Compression: NONE 
Expansion: NONE 
No ALPN negotiated 
SSL-Session: 
    Protocol  : TLSv1.2 
    Cipher    : 0000 
    Session-ID:  
    Session-ID-ctx:  
    Master-Key:  
    Key-Arg   : None 
    PSK identity: None 
    PSK identity hint: None 
    SRP username: None 
    Start Time: 1600987862 
    Timeout   : 300 (sec) 
    Verify return code: 0 (ok) 

Istio ingress gateway logs:

"- - -" 0 - "-" "-" 298 0 1069 - "-" "-" "-" "-" "192.168.101.136:443" outbound|443||myservice.mynamespace.svc.cluster.local 192.168.115.141:42350 192.168.115.141:15443 192.168.125.206:23298 myservice.mynamespace - 

where 192.168.101.136 is the IP of the myservice pod and 192.168.115.141 is the IP of the ingressgateway pod.

Based on the IPs, this means the client connection reached the gateway, the gateway seems to have applied the virtualservice route and logged that it was forwarding this to the pod. Seems normal, except the istio-proxy on the pod shows no activity nor the server logs (though the server doesn't log stuff happening at the transport layer).

AFAIK the server is properly configured for TLS as the following port-forwarded TLS handshake succeeds:

kubectl port-forward -n mynamespace service/myservice 4430:443 &
openssl s_client \
        -connect localhost:4430 \
        -CAfile /path/to/ca.cert \ 
        -cert /path/to/cert.pem \ 
        -key /path/to/cert.key <<< "Q"
# I get back a TLS session ID, looks good.

So this points to a problem with istio's gateway or virtualservice configuration.

Gatway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mygateway
  namespace: mynamespace
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 15443
        name: tls-passthrough
        protocol: TLS
      tls:
        mode: PASSTHROUGH
      hosts:
      - "*"

Virtual Service:

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: myservice
  namespace: mynamespace
spec:
  hosts:
  - "*" 
  gateways:
  - mygateway
  tls:
    - match:
      - port: 15443
        sniHosts:
        - myservice.mynamespace
      route:
      - destination:
          host: myservice
          port:
            number: 443

Kubernetes Service:

apiVersion: v1
kind: Service
metadata:
  name: myservice
  namespace: mynamespace
  labels:
    app: myservice
spec:
  selector:
    app: myservice
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443
      name: grpc-svc

UPDATE: Actually the TLS traffic from the client does reach the server pod, I've confirmed this by doing tcpdump port 443 on the server pod and seeing packets when I run the openssl s_client command. Unclear why istio-proxy on the pod didn't show this, that doesn't explain why the handshake fails.

I noticed something else. Passing -msg flag to openssl s_client, I see nothing coming back, after ">>>" there's no "<<<", yet the tcpdump shows the server pod sending packets back to the gateway.

Answer 1

There were 2 bugs in my configuration:

• in the configuration of my Kubernetes service. Unfortunately the name of my TCP port 443 was grpc-svc and that breaks TLS passthrough. Renaming this port to tcp-svc resolves the problem.

• I should not be using ingress port 15443, that seems to be reserved for something else. Opening another port 9444 on the ingressgateway, then configuring port 9444 on the gateway exactly as I was configuring port 15443 in my question (i.e. config for TLS passthrough), and then configuring the virtual service to route 9444 exactly as I was configuring the virtual service route for 15433 in my question.

Doing both of these allows openssl s_client from an external client to succeed a TLS handshake to a kubernetes service via the ingress.