kubernetes fails to pull a private image [Google Cloud Container Registry, Digital Ocean]

Question

I'm trying to setup GCR with kubernetes

and getting Error: ErrImagePull Failed to pull image "eu.gcr.io/xxx/nodejs": rpc error: code = Unknown desc = Error response from daemon: pull access denied for eu.gcr.io/xxx/nodejs, repository does not exist or may require 'docker login'

Although I have setup the secret correctly in the service account, and added image pull secrets in the deployment spec

deployment.yml

apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose convert
    kompose.version: 1.18.0 (06a2e56)
  creationTimestamp: null
  labels:
    io.kompose.service: nodejs
  name: nodejs
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: nodejs
    spec:
      containers:
      - env:
        - name: MONGO_DB
          valueFrom:
            configMapKeyRef:
              key: MONGO_DB
              name: nodejs-env
        - name: MONGO_HOSTNAME
          value: db
        - name: MONGO_PASSWORD
          valueFrom:
            secretKeyRef:
              name: mongo-secret
              key: MONGO_PASSWORD
        - name: MONGO_PORT
          valueFrom:
            configMapKeyRef:
              key: MONGO_PORT
              name: nodejs-env
        - name: MONGO_USERNAME
          valueFrom:
            secretKeyRef:
              name: mongo-secret
              key: MONGO_USERNAME
        image: "eu.gcr.io/xxx/nodejs"
        name: nodejs
        imagePullPolicy: Always
        ports:
        - containerPort: 8080
        resources: {}
      imagePullSecrets:
        - name: gcr-json-key
      initContainers:
        - name: init-db
          image: busybox
          command: ['sh', '-c', 'until nc -z db:27017; do echo waiting for db; sleep 2; done;']
      restartPolicy: Always
status: {}

used this to add the secret, and it said created

kubectl create secret docker-registry gcr-json-key --docker-server=eu.gcr.io  --docker-username=_json_key  --docker-password="$(cat mycreds.json)"   [email protected]

How can I debug this, any ideas are welcome!

Just be careful you don't have an errant newline character in your password file. — Belly Buster

Shafiq I Shafiq I · Accepted Answer · 2019-10-02T22:21:03

It looks like the issue is caused by lack of permission on the related service account [email protected] which is missing Editor role.

Also,we need to restrict the scope to assign permissions only to push and pull images from google kubernetes engine, this account will need storage admin view permission which can be assigned by following the instructions mentioned in this article [1].

Additionally, to set the read-write storage scope when creating a Google Kubernetes Engine cluster, use the --scopes option to mention this scope "storage-rw"[2].

[1] https://cloud.google.com/container-registry/docs/access-control [2]https://cloud.google.com/container-registry/docs/using-with-google-cloud-platform#google-kubernetes-engine”

kubernetes fails to pull a private image [Google Cloud Container Registry, Digital Ocean]

2 Answers