Why the ambiguous outbound access from GKE private cluster?

Question

I recently created a private GKE kubernetes cluster for running web services and discovered that it's quite locked down and isn't supposed to have any outbound internet access, but has access to GCP services. I'm happy to live with pushing container images to private GCP repo.

However what I do find strange is that after installing some public domain helm charts, some images are pulled from docker hub and other public registries and some are not.

I'm using pre-emptible nodes, so some charts which I had previously deployed have had underlying nodes replaced and the replacements show image pull errors.

Is this due to the multi-tenant nature of the GKE service? Maybe some hosts may have already cached images and so are not actually pulling images?

One example

mongo:3.6

was hanging for over 24 hours for one pod, then eventually was pulled by three pods, but it's a docker hub reference

Answer 1

So it looks like Google mirrors many of the popular public repos. This explains why many of the more common public images can be pulled even without internet access, you're basically just pulling from Google's repo (which you access through private access to APIs).

I'm guessing certain images aren't being mirrored and those ones are the ones hanging.