Push to google container registry fails: Retrying

Question

I'm trying to push to the Google container registry from my Jenkins. The builds run inside the Kubernetes Jenkins Plugin, which uses the gcr.io/cloud-solutions-images/jenkins-k8s-slave to build the docker image into the Kubernetes native Docker.

After authenticating to the Google container registry I'm trying to push the newly built image. This is my pipeline script:

def imageTag = 'gcr.io/project-id/tag'

def version = version from pom 

sh './mvnw package'

sh "docker build -t $imageTag:$version ."

sh('gcloud auth activate-service-account --key-file=$FILE')

sh('docker login -p $(gcloud auth print-access-token) -u _token https://gcr.io')

sh("gcloud docker -- push $imageTag:$version")

The push fails with the following output:

c6ff94654483: Preparing
209db64c273a: Preparing
762429e05518: Preparing
2be465c0fdf6: Preparing
5bef08742407: Preparing
c6ff94654483: Retrying in 5 seconds
5bef08742407: Retrying in 5 seconds
209db64c273a: Retrying in 5 seconds
2be465c0fdf6: Layer already exists
762429e05518: Layer already exists
c6ff94654483: Retrying in 4 seconds
5bef08742407: Retrying in 4 seconds
209db64c273a: Retrying in 4 seconds
c6ff94654483: Retrying in 3 seconds
5bef08742407: Retrying in 3 seconds
209db64c273a: Retrying in 3 seconds
c6ff94654483: Retrying in 2 seconds
5bef08742407: Retrying in 2 seconds
209db64c273a: Retrying in 2 seconds
c6ff94654483: Retrying in 1 second
5bef08742407: Retrying in 1 second
209db64c273a: Retrying in 1 second
5bef08742407: Retrying in 10 seconds
...
unexpected EOF

I've been experiencing the same issue the past few days attempting to setup Gitlab's CI/CD runner to automatically login to GCR when pushing/pulling images to the private repositories. What user account are you logged in with? A personal one, or a service account? Does it have the appropriate IAM permissions? — Don Spaulding
I'm using a service account with it's json file and didn't modify it's permissions. Maybe it doesn't have write permissions to the bucket. I'll check that later and let you know. — Jacob
I just read through your pipeline. Is there a reason you're passing _token as the username? The advanced authentication page seems to say to use oauth2accesstoken? cloud.google.com/container-registry/docs/… — Don Spaulding
Feel free to reach out to the GCR team directly. Contact info here: cloud.google.com/container-registry/docs/support/… — jonjohnson

Don Spaulding Don Spaulding · Accepted Answer · 2017-11-06T15:57:22

The root cause of this issue is that your docker daemon is not authenticated with the credentials necessary to push to gcr.io. For the original question, I believe this is likely because the user account being used was _token instead of oauth2accesstoken.

I was experiencing an error similar to this, except that instead of using docker login, I was using docker-credential-gcr and was getting the same unexpected EOF error.

My problem was the fact that I was running on GCE, from which docker-credential-gcr was detecting and using a different service account via the GCE metadata API.

So, for others experiencing this issue who are running on GCP and trying to authenticate a service account via docker-credential-gcr, you need to tell it to only look at the gcloud credentials, instead of looking at the environment for the metadata API details. My flow looks like this now:

gcloud auth activate-service-account --key-file=$FILE

docker-credential-gcr configure-docker --token-source="gcloud"

docker push gcr.io/....

Hope it helps someone.

Push to google container registry fails: Retrying

5 Answers