k8s - Failed to pull image....Error response from daemon: pull access denied for {private_repo}, repository does not exist

Question

Getting this message:

Failed to pull image....Error response from daemon: pull access denied for {private_repo}, the repository does not exist or may require 'docker login'

After deploying new helm chart using AWS ECR BUT

1. Full private repo path is correct and image exists in ECR, in ReplicationController: private_repo/serviceXYZ:latest-develop
2. Other pods using the SAME repo but different paths ARE working, ex: private_repo/serviceABC (their latest local images are several months old and we did deploy them recently which tells me we don't pull them locally but straight from ECR)
3. ~/.docker/config.json shows that it's logged in
4. There is NO secret in other services (no imagePullSecrets) which are pulled successfully

Any thoughts appreciated.

Answer 1

You need to authenticate to ECR to pull image. If you haven't done so, follow instructions here. Basically you get an authorization token from AWS to pass it to docker login. The account required by ECR is IAM-based and different from your local Docker account.

If you have already done that, the token may have expired. Log in again then.

The reason you don't have to do this for other pods is likely those images have been built or pulled to local so Docker doesn't have to download it (with the imagePullPolicy of the pod set to IfNotPresent as default).

Answer 2

You didn't specify your Kubernetes resource (i.e Pod, Deployment, etc). But it's most likely as far as I can tell because either:

• You are missing the ImagePullSecrets in the specific Pod definition:

  kubectl create secret generic regcred \
    --from-file=.dockerconfigjson=~/.docker/config.json> \
    --type=kubernetes.io/dockerconfigjson
  

  Then the pod:

  apiVersion: v1
  kind: Pod
  metadata:
    name: myservice
  spec:
    containers:
    - name: yourXYZservice
      image: serviceXYZ:latest-develop
    imagePullSecrets:
    - name: regcred
  

• Your private ECR repo doesn't have latest-develop tag. Before pushing, tag your image and push it with the tag:

  docker tag <image-id> serviceXYZ:latest-develop
  docker push serviceXYZ:latest-develop
  

Answer 3

The issue was with permissions in ECR for this particular repo, I put following permissions with "Effect": "Allow" and it worked:

"Action": [
    "ecr:BatchCheckLayerAvailability",
    "ecr:BatchGetImage",
    "ecr:CompleteLayerUpload",
    "ecr:DescribeImages",
    "ecr:GetDownloadUrlForLayer",
    "ecr:InitiateLayerUpload",
    "ecr:ListImages",
    "ecr:PutImage",
    "ecr:UploadLayerPart"
  ]