Different Firewall Rules for Kubernetes Cluster

1
votes

I am running some internal services and also some customer facing services in one K8s cluster. The internal ones should only be accessible from some specific ips and the customer facing services should be accessible worldwide.

So I created my Ingresses and an nginx Ingress Controller and some K8s LoadBalancer Services with the proper ip filters.

Now I see those Firewall rules in GCP are created behind the scenes. But they are conflicting and the "customer facing" firewall rules overrule the "internal" ones. And so everything of my K8s Cluster is visible worldwide.

The usecase sounds not that exotic to me - do you have an idea how to get some parts of a K8s cluster protected by firewall rules and some accessible everywhere?

1
kubernetesgoogle-cloud-platform
With "internal", you mean accessible from your cluster other pods? or you mean accessible from your on-premise network? or accessible from your GCP project? - suren
I mean these services need to be accessible from our on-premise network. - Hubert Ströbitzer
What are the ranges of your GCP and on-premise networks? Do they overlap? - suren
No, they do not overlap. In the meantime I found a solution. If I deploy 2 different ingress, ingress-controllers, loadbalancers with different firewall settings operating on different ports I can achieve my goals... but with the drawback of having some non standard port, eg myservice.mydomain.wtf:12345 - Hubert Ströbitzer
You shouldn't be able to use 2 ingress controllers. - suren

1 Answers

2
votes

As surprising as it is, the L7 (http/https) load balancer in GCP created by a Kubernetes Ingress object has no IP whitelisting capabilities by default, so what you described is working as intended. You can filter on your end using the X-Forwarded-For header (see Target Proxies under Setting Up HTTP(S) Load Balancing).

Whitelisting will be available trough Cloud Armour, which is in private beta at the moment.

To make this situation slightly more complicated: the L4 (tcp/ssl) load balancer in GCP created by a Kubernetes LoadBalancer object (so, not an Ingress) does have IP filtering capability. You simply set .spec.loadBalancerSourceRanges on the Service for that. Of course, a Service will not give you url/host based routing, but you can achieve that by deploying an ingress controller like nginx-ingress. If you go this route you can still create Ingresses for your internal services you just need to annotate them so the new ingress controller picks them up. This is a fairly standard solution, and is actually cheaper than creating L7s for each of your internal services (you will only have to pay for 1 forwarding rule for all of your internal services).

(By "internal services" above I meant services you need to be able to access from outside of the itself cluster but only from specific IPs, say a VPN, office, etc. For services you only need to access from inside the cluster you should use type: ClusterIP)