How to specify docker credentials for Kubernetes static pods with CRI enabled?

Question

When starting up a Kubernetes cluster, I load etcd plus the core kubernetes processes - kube-proxy, kube-apiserver, kube-controller-manager, kube-scheduler - as static pods from a private registry. This has worked in the past by ensuring that the $HOME environment variable is set to "/root" for kubelet, and then having /root/.docker/config.json defined with the credentials for the private docker registry.

When attempting to run Kubernetes 1.6, with CRI enabled, I get errors in the kubelet log saying it cannot pull the pause:3.0 container from my private docker registry due to no authentication.

Setting --enable-cri=false on the kubelet command line works, but when CRI is enabled, it doesn't seem to use the /root/.docker/config file for authentication.

Is there some new way to provide the docker credentials needed to load static pods when running with CRI enabled?

Answer 1

In 1.6, I managed to make it work with the following recipe in https://kubernetes.io/docs/concepts/containers/images/#specifying-imagepullsecrets-on-a-pod

$ kubectl create secret docker-registry myregistrykey --docker-server=DOCKER_REGISTRY_SERVER --docker-username=DOCKER_USER --docker-password=DOCKER_PASSWORD --docker-email=DOCKER_EMAIL

You need to specify newly created myregistrykey as the credential under imagePullSecrets field in the pod spec.

apiVersion: v1
kind: Pod
metadata:
  name: foo
  namespace: awesomeapps
spec:
  containers:
    - name: foo
      image: janedoe/awesomeapp:v1
  imagePullSecrets:
    - name: myregistrykey

Answer 2

It turns out that there is a deficiency in the CRI capabilities in Kubernetes 1.6. With CRI, the "pause" container - now called the "Pod Sandbox Image" is treated as a special case - because it is an "implementation detail" of the container runtime whether or not you even need it. In the 1.6 release, the credentials applied for other containers, from /root/.docker/config.json, for example, are not used when trying to pull the Pod Sandbox Image.

Thus, if you are trying to pull this image from a private registry, the CRI logic doesn't associate the credentials with the pull request. There is now a Kubernetes issue (#45738) to address this, targeted for 1.7.

In the meantime, an easy workaround is to pre-pull the "Pause" container into the node's local docker image cache before starting up the kubelet process.