How to debug ImagePullBackOff on gcr.io

Question

I can docker push and pull to gcr.io

Update: Same thing if I use Google kubernetes engine site interface and the deploy button.

Update: Upgraded the the cluster from looking at this, did not work: Kubernetes pods failing on "Pod sandbox changed, it will be killed and re-created"

Update: To fully reproduce this with httpbin:

gcloud auth configure-docker
docker pull kennethreitz/httpbin
docker tag kennethreitz/httpbin:latest gcr.io/<proj-id>/images/httpbin:v1.0
docker push gcr.io/<proj-id>/images/httpbin:v1.0
kubectl run test --image=gcr.io/<proj-id>/images/httpbin:v1.0

Then I get ImagePullBackOff:

kubectl describe pod test-5f447f48f4-kcpg9
Name:           test-5f447f48f4-kcpg9
Namespace:      default
Node:           <proj-id>-cluster-default-pool-f398e01b-5kmb/10.132.0.2
Start Time:     Tue, 28 Aug 2018 09:08:43 +0200
Labels:         pod-template-hash=1900390490
                run=test
Annotations:    kubernetes.io/limit-ranger=LimitRanger plugin set: cpu request for container test
Status:         Pending
IP:             10.16.1.46
Controlled By:  ReplicaSet/test-5f447f48f4
Containers:
  test:
    Container ID:   
    Image:          gcr.io/<proj-id>/images/httpbin:v1.0
    Image ID:       
    Port:           <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:        100m
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-9mcms (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  default-token-9mcms:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-9mcms
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                 Age                From                                                   Message
  ----     ------                 ----               ----                                                   -------
  Normal   Scheduled              5m                 default-scheduler                                      Successfully assigned test-5f447f48f4-kcpg9 to <proj-id>-cluster-default-pool-f398e01b-5kmb
  Normal   SuccessfulMountVolume  5m                 kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  MountVolume.SetUp succeeded for volume "default-token-9mcms"
  Normal   SandboxChanged         5m (x3 over 5m)    kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  Pod sandbox changed, it will be killed and re-created.
  Normal   BackOff                5m (x6 over 5m)    kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  Back-off pulling image "gcr.io/<proj-id>/images/httpbin:v1.0"
  Normal   Pulling                4m (x3 over 5m)    kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  pulling image "gcr.io/<proj-id>/images/httpbin:v1.0"
  Warning  Failed                 4m (x3 over 5m)    kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  Failed to pull image "gcr.io/<proj-id>/images/httpbin:v1.0": rpc error: code = Unknown desc = Error response from daemon: repository gcr.io/<proj-id>/images/httpbin not found: does not exist or no pull access
  Warning  Failed                 4m (x3 over 5m)    kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  Error: ErrImagePull
  Warning  Failed                 31s (x23 over 5m)  kubelet, <proj-id>-cluster-default-pool-f398e01b-5kmb  Error: ImagePullBackOff

Update: Same thing running in Google Cloud Shell.

In the Cloud Shell: docker images, also shows the image

Update: I can see the image in the browser console(https://console.cloud.google.com/gcr/images/projId/), and if I copy the path it is the same as the beneath - just with another project id.

But kubectl run aName --image=gcr.io/projectid/myimages/myimage:v1.0

Is failing to pull the image on Google Kubernetes Engine

ImagePullBackOff

kubectl describe pod aName...

Failed to pull image "myimages/myimage:v1.0": rpc error: code = Unknown desc = Error response from daemon: repository myimages/myimage not found: does not exist or no pull access

So it could be "or no pull access" but how do I fix that?

Ahmet Alp Balkan Ahmet Alp Balkan · Accepted Answer · 2018-08-27T21:36:59

The myimages/myimage format suggests you're using Docker Hub. If so, you may need to authenticate to Docker Hub using an image pull secret (https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/#create-a-secret-in-the-cluster-that-holds-your-authorization-token).

If you're using Google Container Registry (GCR, which has images starting with gcr.io/*), GKE nodes will already have access to the GCR images in the same GCP project as the cluster –you don't need to do anything. (However, if someone has created GKE cluster or node pool with less-than-default --scopes , such as omitting storage-ro, then your cluster will fail to pull images from the GCR in the same GCP project.)

Follow the tutorial at https://cloud.google.com/kubernetes-engine/docs/tutorials/hello-app to:

create a cluster
build an image from source code
push it to GCR
deploy to GKE.