how to terminate ssl at ingress-gateway in istio?

2
votes

I am trying to experiment ssl connection in istio ingress gateway.

From here istio ssl gateway without termination, i assume that istio ingress gateway by default should terminate ssl.

I have installed istio with demo profile, via istioctl. I have also installed my service svc1.

Apart from these, below are what my resources are with routng logic:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: vs-gateway
  namespace: myns
spec:
  selector:
    istio: ingressgateway # use istio default controller
  servers:
  - port:
      number: 443
      name: https
      protocol: HTTPS
    tls:
      mode: SIMPLE
      serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
      privateKey: /etc/istio/ingressgateway-certs/tls.key
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: vs
  namespace: myns
spec:
  hosts:
  - "*"
  gateways:
  - vs-gateway
  http:
  - match:
    - uri:
        prefix: "/svc1/"
    rewrite:
      uri: "/"
    route:
    - destination:
        host: svc1
        port:
          number: 80

I found the Gateway url via this. For experiment i had also enabled http on ingress gateway. That way curl http://172.17.0.2:<http_node_port> works with 200 response. Later i removed http from ingress gateway and kept only https (as https is my primary goal to be recieved by ingress gateway)

And then trying curl with https on gateway url. I get 503.

$ curl -ivk https://172.17.0.2:<https_node_port>/svc1/user
*   Trying 172.17.0.2...
* TCP_NODELAY set
* Connected to 172.17.0.2 (172.17.0.2) port 30278 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=CA; O=Abc; CN=example.com
*  start date: Dec 31 08:22:32 2019 GMT
*  expire date: Jan 30 08:22:32 2020 GMT
*  issuer: C=US; ST=CA; O=Abc; CN=example.com
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* Using Stream ID: 1 (easy handle 0x55c961626580)
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET /svc1/user HTTP/2
> Host: 172.17.0.2:30278
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/2 503 
HTTP/2 503 
< content-length: 95
content-length: 95
< content-type: text/plain
content-type: text/plain
< date: Thu, 02 Jan 2020 08:13:49 GMT
date: Thu, 02 Jan 2020 08:13:49 GMT
< server: istio-envoy
server: istio-envoy

< 
* Connection #0 to host 172.17.0.2 left intact
upstream connect error or disconnect/reset before headers. reset reason: connection termination

I also enabled sidecar istio proxy debuggin on svc pod sidecar. I got below logs

[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [external/envoy/source/extensions/filters/listener/tls_inspector/tls_inspector.cc:72] tls inspector: new connection accepted
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:30] Called tcp filter: Filter
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:40] Called tcp filter: initializeReadFilterCallbacks
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [external/envoy/source/common/tcp_proxy/tcp_proxy.cc:204] [C88] new tcp proxy session
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:133] [C88] Called tcp filter onNewConnection: remote 10.244.0.5:34148, local 10.244.0.16:3000
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][filter] [external/envoy/source/common/tcp_proxy/tcp_proxy.cc:347] [C88] Creating connection to cluster inbound|80|serviceport|svc1.myns.svc.cluster.local
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:83] creating a new connection
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:364] [C89] connecting
[Envoy (Epoch 0)] [2020-01-02 06:53:19.392][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:711] [C89] connecting to 127.0.0.1:3000
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:720] [C89] connection in progress
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:109] queueing request due to no available connections
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:333] [C88] new connection
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:559] [C89] connected
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:285] [C89] assigning connection
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][filter] [external/envoy/source/common/tcp_proxy/tcp_proxy.cc:541] TCP:onUpstreamEvent(), requestedServerName: 
[Envoy (Epoch 0)] [2020-01-02 06:53:19.393][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:143] Called tcp filter completeCheck: OK
[Envoy (Epoch 0)] [2020-01-02 06:53:19.395][28][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
  key: "connection.event"
  value {
    string_value: "open"
  }
}
attributes {
  key: "connection.id"
  value {
    string_value: "38a9b348-1730-4e0b-9664-fbbaeedd9215-88"
  }
[Envoy (Epoch 0)] [2020-01-02 06:53:19.395][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:100] [C88] Called tcp filter onRead bytes: 664
[Envoy (Epoch 0)] [2020-01-02 06:53:19.396][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:123] [C88] Called tcp filter onWrite bytes: 28
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:123] [C88] Called tcp filter onWrite bytes: 0
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:100] [C88] Called tcp filter onRead bytes: 34
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:527] [C88] remote close
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C88] closing socket: 0
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:174] [C88] Called tcp filter onEvent: 0 upstream 127.0.0.1:3000
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
  key: "connection.duration"
  value {
    duration_value {
      nanos: 6151000
    }
  }
}
attributes {
  key: "connection.event"
  value {
    string_value: "close"
  }
}
at
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C89] closing data_to_write=34 type=0
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:88] [C88] adding to cleanup list
[Envoy (Epoch 0)] [2020-01-02 06:53:19.398][28][debug][filter] [src/envoy/tcp/mixer/filter.cc:35] Called tcp filter : ~Filter
[Envoy (Epoch 0)] [2020-01-02 06:53:19.399][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:589] [C89] write flush complete
[Envoy (Epoch 0)] [2020-01-02 06:53:19.399][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C89] closing socket: 1
[Envoy (Epoch 0)] [2020-01-02 06:53:19.399][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:124] [C89] client disconnected
[Envoy (Epoch 0)] [2020-01-02 06:53:19.399][28][debug][pool] [external/envoy/source/common/tcp/conn_pool.cc:238] [C89] connection destroyed

With the logs, it seems like ingress gateway is forwarding ssl request to svc. (please correct here if i am wrong)

So in the end, can anyone please help me out getting ssl terminated at ingress gateway, and forward plain http request to svc?

1
sslroutingtls1.2istiogateway
Which version You try to make, with tls mode: PASSTHROUGH or SIMPLE? PASSTHROUGH will work on nginx side, like in istio documentation provided by You, SIMPLE will work on istio side. Common issue when using mode: SIMPLE is destination rule which must include the trafficPolicy, since You use tls in your gateway. More about it here. Let me know which mode You want to use and if the link solved your problem. – Jakub
this is what i get with istioctl auth svc1.myns.svc.cluster.local:80 OK PERMISSIVE - /default - , i have not defined trafficPolicy, and i have not configured mutual TLS, its just one way TLS – codeofnode
Also there is no any destination rule, hence trafficePolicy is not defined anyway. – codeofnode
Could You please add yaml of your svc1 deployment,svc? – Jakub
This is my svc1 github.com/mock-server/mockserver/blob/master/helm/mockserver/… – codeofnode

1 Answers

0
votes

Based on this github issue and this istio documentation

Named service ports: Service ports must be named. The port name key/value pairs must have the following syntax: name: [-]. See Protocol Selection for more details.

Based on mockserver service 

ports:
    - name: serviceport

I would recommend to change it to http/https like mentioned there and confirmed by community member who had same issue on github.

Manual protocol selection

Protocols can be specified manually by naming the Service port name: [-]. The following protocols are supported:

  • grpc
  • grpc
  • web
  • http
  • http2
  • https
  • mongo
  • mysql*
  • redis*
  • tcp
  • tls
  • udp

*These protocols are disabled by default to avoid accidentally enabling experimental features. To enable them, configure the corresponding Pilot environment variables.