Istio Egresses with Kubernetes Services

Question

(Using Istio 0.5.1, kubectl 1.9.1/1.9.0 for client/server, minikube 0.25.0)

I'm trying to get Istio EgressRules to work with Kubernetes Services, but having some trouble.

I tried to set up EgressRules 3 ways:

An ExternalName service which points to another domain (like www.google.com)
A Service with no selector, but an associated Endpoint object (for services that have an IP address but no DNS name)
(for comparison) No Kubernetes service, just an EgressRule

I figured I could use the FQDN of the kubernetes service as the HTTP-based EgressRule destination service (like ext-service.default.svc.cluster.local), and this is what I attempted for both an ExternalName service as well as a Service with no selectors but an associated Endpoints object.

For the former, I created the following yaml file:

kind: Service
apiVersion: v1
metadata:
  name: ext-service
spec:
  type: ExternalName
  externalName: www.google.com
---
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: ext-egress-rule
spec:
  destination:
    service: ext-service.default.svc.cluster.local
  ports:
    - port: 443
      protocol: https

For the latter, I created this yaml file (I just pinged google and grabbed the IP address):

kind: Endpoints
apiVersion: v1
metadata:
  name: ext-service
subsets:
  - addresses:
      - ip: 216.58.198.78
    ports:
      - port: 443
---
kind: Service
apiVersion: v1
metadata:
  name: ext-service
spec:
  ports:
  - protocol: TCP
    port: 443
    targetPort: 443
---
apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: ext-service-egress-rule
spec:
  destination:
    service: ext-service.default.svc.cluster.local
  ports:
    - port: 443
      protocol: https

In both cases, in the application code, I access:

http://ext-service.default.svc.cluster.local:443

My assumption is that the traffic will flow like:

[[ app -> envoy proxy -> (tls origination) -> kubernetes service ]] -> external service

where [[ ... ]] is the boundary of the service mesh (and also the Kubernetes cluster)

Results:

The ExternalName Service almost worked as expected, but it brought me to Google's 404 page (and sometimes the response just seemed empty, not sure how to replicate one or the other specifically)
The Service with the Endpoint object did not work, instead printing this message (when making the request via Golang, but I don't think that matters):

Get http://ext-service.default.svc.cluster.local:443: EOF

This also sometimes gives an empty response.

I'd like to use Kubernetes services (even though it's for external traffic) for a few reasons:

You can't use an IP address for the EgressRule's destination service. From Egress Rules configuration: "The destination of an egress rule ... can be either a fully qualified or wildcard domain name".
For external services that don't have a domain name (some on-prem legacy/monolith service without a DNS name), I'd like the application to be able to access them not by IP address but by a kube-dns (or Istio-related similar) name.
(related to previous) I like the additional layer of abstraction that a Kubernetes service provides, so I can change the underlying destination without changing the EgressRule (unless I'm mistaken and this isn't the right way to architect this). Is the EgressRule meant to replace Kubernetes services for external traffic entirely and without creating additional Kubernetes services?

Using https:// in the app code isn't an option because then the request would have to disable TLS verification since the kube-dns name doesn't match any on the certificate. It also wouldn't be observable.

If I use the following EgressRule (without any Kubernetes Services), accessing Google via http://www.google.com:443 works fine, getting the exact html representation that I expect:

apiVersion: config.istio.io/v1alpha2
kind: EgressRule
metadata:
  name: google-egress-rule
spec:
  destination:
    service: www.google.com
  ports:
    - port: 443
      protocol: https

I saw there's a TCP EgressRule, but I would rather not have to specify rules for each block of IPs. From TCP Egress: "In TCP egress rules as opposed to HTTP-based egress rules, the destinations are specified by IPs or by blocks of IPs in CIDR notation.".

Also, I would still like the HTTP-based observability that comes from L7 instead of L4, so I'd prefer an HTTP-based egress. (With TCP Egresses, "The HTTPS traffic originated by the application will be treated by Istio as opaque TCP").

Any help getting a Kubernetes service as "destination service" of an EgressRule (or help understanding why this isn't necessary if that's the case) is appreciated. Thanks!

Vadim Eisenberg Vadim Eisenberg · Accepted Answer · 2018-03-14T03:22:10

The solution is:

Define a Kubernetes ExternalName service to point to www.google.com
Do not define any EgressRules
Create a RouteRule to set the Host header.

In your case, define an ExternalName service with the port and the protocol:

kind: Service
apiVersion: v1
metadata:
  name: ext-service
spec:
  type: ExternalName
  externalName: www.google.com
   ports:
   - port: 80
   # important to set protocol name
   name: http
---

Define an HTTP Rewrite Route Rule to set the Host header:

apiVersion: config.istio.io/v1alpha2
kind: RouteRule
metadata:
  name: externalgoogle-rewrite-rule
  #namespace: default
spec:
  destination:
    name: ext-service
  rewrite:
    authority: www.google.com
---

Then access it with curl, for example: curl ext-service

Without the Route Rule, the request will arrive to google.com, with the Host header being ext-service. The web server does not know where to forward such a request since google.com does not have such a virtual host. This is what you experienced:

it brought me to Google's 404 page

Istio Egresses with Kubernetes Services

1 Answers