GKE/Istio: outside world cannot connect to service in private cluster

2
votes

I've created a private GKE cluster with Istio through the Cloud Console UI. The cluster is set up with VPC Peering to be able to reach another private GKE cluster in another Google Cloud Project.

I've created a Deployment (called website) with a Service in Kubernetes in the staging namespace. My goal is to expose this service to the outside world with Istio, using the Envoy proxy. I've created the necessary VirtualService and Gateway to do so, following this guide.

When running "kubectl exec ..." to access a pod in the private cluster, I can successfully connect to the internal IP address of the website service, and see the output of that service with "curl".

I have set up a NAT Gateway so pods in the private cluster can connect to the Internet. I confirmed this by curl-ing various non-Google web pages from within the website pod.

However, I can't connect to the website service from the outside, using the External IP of the istio-ingressgateway service, as the guide above mentions. Instead, curl-ing that External IP leads to a timeout.

I've put the full YAML config for all related resources in a private Gist, here: https://gist.github.com/marceldegraaf/0f36ca817a8dba45ac97bf6b310ca282

I'm wondering if I'm missing something in my config here, or if my use case is actually impossible?

3
kubernetesgoogle-kubernetes-engineistio
If you're just using external IP of a node, rather than a LoadBalancer or Ingress, you'll need to make sure your GCE firewall rules allow the traffic. - Paul Annetts
@PaulAnnetts thanks for your reply. The External IP of the istio-ingressgateway is bound to a forwarding rule in Cloud Load Balancer, with the GKE nodes as the Target Pool. I've added a Firewall Rule to explicitly allow all incoming traffic on TCP:80 for the entire network, but even then my curl to the External IP results in a timeout. Any idea what I'm missing? - Marcel de Graaf
Did you check the logs for istio-ingressgateway? If your service is getting hit and if the service has some error, it would show in that log. - mjkool
Thanks for your reply @mjkool. When checking the logs for istio-ingressgateway I see no requests coming in when trying to access the External IP from outside. I do see requests in the logs of istio-ingressgateway when I try to curl it from inside the cluster. - Marcel de Graaf
@MarceldeGraaf, any solution to the problem? - Renatto Machado

3 Answers

0
votes

Looking at your Gist I suspect the problem lies in the joining up of the Gateway to the istio-ingressgateway.

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: website-gateway
  namespace: staging
  labels:
    version: v1
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"

In particular I'm not convinced the selector part is correct.

You should be able to do something like
kubectl describe po -n istio-system istio-ingressgateway-rrrrrr-pppp
to find out what the selector is trying to match in the Istio Ingress Gateway pod.

0
votes

I had the same problem. On my case, the istio virtual service dont find my service.

Try this on your VirtualService:

   route:
   - destination:
       host: website
       port: 
         number: 80
-1
votes

From verifying all options, the only way to have the private GKE cluster with Istio to be exposed to externally is to use Cloud NAT.

Since the Master node within GKE is a managed service, there are current limits when using Istio with a private cluster. The only workaround that would accomplish your use case is to use Cloud NAT. I have also attached an article on how to get started using Cloud NAT here.