Internet connectivity for GKE nodes

1
votes

I created a GKE cluster with the follwing command:

gcloud container clusters create experiment --num-nodes=1 --network default --subnetwork default --enable-private-nodes --enable-private-endpoint --enable-ip-alias --master-ipv4-cidr 172.16.0.16/28 --no-enable-basic-auth --no-issue-client-certificate

I have no egress rules in my VPC Firewall I have a autocreated default route under VPC routes that applies to the GKE nodes and allows internet access.

On the GKE node I can:

    $ docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bb79b6b2107f: Pull complete 
111447d5894d: Pull complete 
a95689b8e6cb: Pull complete 
1a0022e444c2: Pull complete 
32b7488a3833: Pull complete 
Digest: sha256:ed7f815851b5299f616220a63edac69a4cc200e7f536a56e421988da82e44ed8
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest



docker pull ubuntu
Using default tag: latest
latest: Pulling from library/ubuntu
6a5697faee43: Pull complete 
ba13d3bc422b: Pull complete 
a254829d9e55: Pull complete 
Digest: sha256:fff16eea1a8ae92867721d90c59a75652ea66d29c05294e6e2f898704bdb8cf1
Status: Downloaded newer image for ubuntu:latest
docker.io/library/ubuntu:latest

But I can't:

$ wget https://www.amazon.com
--2020-10-31 19:22:44--  https://www.amazon.com/
Resolving www.amazon.com... 13.226.21.44
Connecting to www.amazon.com|13.226.21.44|:443...

But I can:

  $ wget https://www.google.com
--2020-10-31 19:23:15--  https://www.google.com/
Resolving www.google.com... 172.217.212.147, 172.217.212.99, 172.217.212.106, ...
Connecting to www.google.com|172.217.212.147|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: 'index.html.1'
index.html.1                                   [ <=>                                                                                    ]  12.48K  --.-KB/s    in 0s      
2020-10-31 19:23:15 (72.1 MB/s) - 'index.html.1' saved [12782]





   route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.128.0.1      0.0.0.0         UG    1024   0        0 eth0
10.108.2.0      0.0.0.0         255.255.255.0   U     0      0        0 cbr0
10.128.0.1      0.0.0.0         255.255.255.255 UH    1024   0        0 eth0
169.254.123.0   0.0.0.0         255.255.255.0   U     0      0        0 docker0

Whats happening with internet connectivity on GKE nodes. I can reach docker hub but not www.amazon.com ? Little confused here.

1
google-cloud-platformgoogle-kubernetes-enginegke-networking
Pods (GKE node) need to access stuff that they need to build containers that run inside them. Looks like Google allows only that kind of resources, like dockerhub and their address space (where you might have container registries behind a CDN) and they block everything else. That's the reason you cannot open amazon.com. – Iñigo González
@Iñigo What you said, makes sense. Thanks. But looking for something more definitive and official. – Ankur Agarwal
Add an allow external access rule on port 443 to your firewall – Daniel Lee

1 Answers

3
votes

Whats happening with internet connectivity on GKE nodes. I can reach docker hub but not www.amazon.com ? Little confused here.

I know it may seem somewhat confusing at first look as you may think that you do have access to Docker Hub. Well, in fact you don't.

Did you try to curl https://hub.docker.com/ ? I guess you didn't. If you did, you'd notice that it also fails.

As you can read here:

Nodes in a private cluster do not have outbound access to the public internet. They have limited access to Google APIs and services, including Container Registry.

So, what's actually happening here ?

You're not pulling images directly from Docker Hub, but from a mirror of it, maintained by Google Container Registry. You can check it in a very simple way. If you pull nginx (which equals to nginx:latest) it works perfectly, however if you try to pull let's say nginx:1.14.2 it will fail. This is because GCR doesn't keep older versions of all images available on Docker Hub. It's also mentioned in the official docs:

You cannot fetch images directly from Docker Hub. Instead, use images hosted on Container Registry. Note that while Container Registry's Docker Hub mirror is accessible from a private cluster, it should not be exclusively relied upon. The mirror is only a cache, so images are periodically removed, and a private cluster is not able to fall back to Docker Hub.

I explained it in depth some time ago in this answer so you may also want to take a look at it. It is also well explained in the official docs.

But I can:

$ wget https://www.google.com

Come on, you're on GCP platform 🙂, so you're accessing google.com from within Google's network and this one is probably not the best choice for testing connectivity with public internet on this specific cloud platform.