How to route egress traffic through operator gateway

Question

I have deployed egress-operator in my ubuntu machine and this operator is internally using envoy proxy to control egress traffic.

The idea is to allow only whitelisted domains from test-pod for egress. I have applied the external service yaml of this operator but it's giving the opposite result, instead of allowing google.com its blocking google.com and allowing other calls. What possibly am I doing wrong?

My ExternalService.yaml

    apiVersion: egress.monzo.com/v1
    kind: ExternalService
    metadata:
      name: google
    spec:
      dnsName: google.com
      # optional, defaults to false, instructs dns server to rewrite queries for dnsName
      hijackDns: true
      ports:
      - port: 80
      - port: 443
        protocol: TCP
      minReplicas: 1
      maxReplicas: 3

My testpod.yaml

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: testNs-system
  labels:
    egress.monzo.com/allowed-gateway: google
spec:
  containers:
  - image: nginx:1.14.2
    command:
      - "sleep"
      - "604800"
    imagePullPolicy: IfNotPresent
    name: nginx
  restartPolicy: Always

From testpod when curl -v https://google.com is blocking and other urls are allowed. As per operator's Readme, I need a defaut-deny-Egress K3s policy also, therefore I applied that too. but after default-deny-Egress policy all egress calls including google.com (the one whitelisted) is blocking from testpod.

Default-Deny-All-Egress.yaml

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all-egress
  namespace: testNs-system
spec:
  podSelector:
    matchLabels:
      app: nginx
      egress.monzo.com/allowed-gateway: google
  policyTypes:
  - Egress
  egress: []

How can I route the egress traffic from the egress-operator pod or egress-operator gateway?

Not sure about egress.monzo, but this is possible to achieve using istio egress. Are you tight to any specific solution like monzo? Please refer to istio egress - Accessing External Services — moonkotte
I am restrict to use opensource(free of cost) solutions. is istio free or paid/enterprise solution ? — solveit

Unknown Unknown · Accepted Answer · 2021-07-23T09:43:23

Posting this answer as a community wiki, feel free to edit and expand.

Istio can be used as a solution for this case. It's an open-source project so it doesn't require to pay for its usage.

Istio has a very good documentation with examples how to achieve different results. Documentation is much better for istio in comparison with monzo operator + a lot of big companies use it so it's reliable solution.

Accessing external services and whow it works:

Because all outbound traffic from an Istio-enabled pod is redirected to its sidecar proxy by default, accessibility of URLs outside of the cluster depends on the configuration of the proxy. By default, Istio configures the Envoy proxy to pass through requests for unknown services. Although this provides a convenient way to get started with Istio, configuring stricter control is usually preferable.

Please find istio documentation and covered use cases with the same goal as yours:

How to route egress traffic through operator gateway

1 Answers