Kubernetes POD Timeout expired waiting for volumes to attach/mount

Question

I can't mount GCE PersistentVolumes using Kubernetes 1.8.0, each POD are stuck in ContainerCreating state.

This output is from a test environment I put up for this lab. Worth to mention is that I'm using Compute Engine, NOT Kubernetes Engine.

I have not configured any cloud settings and I wounder if this might be the root cause but gcloud works perfectly fine from the worker and all my VMs in this lab environment are allowed full access to the API.

Error message on the worker

Jan  2 13:03:58 worker-0 kubelet[1421]: E0102 13:03:58.733299    1421 kubelet.go:1628] Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]; skipping pod

POD description

bofh:~$ kubectl describe pod mysql-cgui-01-5c85f7dd86-gt2s8
Name:           mysql-cgui-01-5c85f7dd86-gt2s8
Namespace:      default
Node:           worker-0/10.240.0.20
Start Time:     Tue, 02 Jan 2018 12:15:49 +0000
Labels:         name=mysql-cgui-01
                pod-template-hash=1741938842
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"mysql-cgui-01-5c85f7dd86","uid":"ab10f9ef-efb6-11e7-a385-42010af...
Status:         Pending
IP:
Created By:     ReplicaSet/mysql-cgui-01-5c85f7dd86
Controlled By:  ReplicaSet/mysql-cgui-01-5c85f7dd86
Containers:
  mysql-cgui-01:
    Container ID:
    Image:          external/mysql:latest
    Image ID:
    Port:           3306/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Mounts:
      /data/mysql from mysql-cgui-01 (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-tb6sm (ro)
Conditions:
  Type           Status
  Initialized    True
  Ready          False
  PodScheduled   True
Volumes:
  mysql-cgui-01:
    Type:       GCEPersistentDisk (a Persistent Disk resource in Google Compute Engine)
    PDName:     mysql-cgui-01
    FSType:     ext4
    Partition:  0
    ReadOnly:   false
  default-token-tb6sm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-tb6sm
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason                 Age                From               Message
  ----     ------                 ----               ----               -------
  Normal   Scheduled              55m                default-scheduler  Successfully assigned mysql-cgui-01-5c85f7dd86-gt2s8 to worker-0
  Normal   SuccessfulMountVolume  55m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
  Warning  FailedMount            41m (x6 over 53m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
  Warning  FailedSync             41m (x6 over 53m)  kubelet, worker-0  Error syncing pod
  Normal   SuccessfulMountVolume  38m                kubelet, worker-0  MountVolume.SetUp succeeded for volume "default-token-tb6sm"
  Warning  FailedMount            4m (x15 over 36m)  kubelet, worker-0  Unable to mount volumes for pod "mysql-cgui-01-5c85f7dd86-gt2s8_default(ab17eaf2-efb6-11e7-a385-42010af0000a)": timeout expired waiting for volumes to attach/mount for pod "default"/"mysql-cgui-01-5c85f7dd86-gt2s8". list of unattached/unmounted volumes=[mysql-cgui-01]
  Warning  FailedSync             2m (x16 over 36m)  kubelet, worker-0  Error syncing pod

Testing using gcloud from worker-0

worker-0:~$ gcloud compute disks list
NAME           ZONE            SIZE_GB  TYPE         STATUS
bofh           europe-west1-d  20       pd-standard  READY
controller-0   europe-west1-c  200      pd-standard  READY
controller-1   europe-west1-c  200      pd-standard  READY
controller-2   europe-west1-c  200      pd-standard  READY
mysql-cgui-01  europe-west1-c  10       pd-standard  READY
mysql-cgui-02  europe-west1-c  10       pd-standard  READY
worker-0       europe-west1-c  200      pd-standard  READY
worker-1       europe-west1-c  200      pd-standard  READY
worker-2       europe-west1-c  200      pd-standard  READY

Worker-0 kubelet flags

ExecStart=/usr/local/bin/kubelet \
  --allow-privileged=true \
  --anonymous-auth=false \
  --authorization-mode=Webhook \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --cluster-dns=10.32.0.10 \
  --cluster-domain=cluster.local \
  --container-runtime=docker \
  --image-pull-progress-deadline=2m \
  --kubeconfig=/var/lib/kubelet/kubeconfig \
  --network-plugin=cni \
  --pod-cidr=10.200.0.0/24 \
  --register-node=true \
  --require-kubeconfig \
  --runtime-request-timeout=15m \
  --tls-cert-file=/var/lib/kubelet/worker-0.pem \
  --tls-private-key-file=/var/lib/kubelet/worker-0-key.pem \
  --cloud-provider=gce \
  --v=2

Worker-0 kube-proxy flags

ExecStart=/usr/local/bin/kube-proxy \
  --cluster-cidr=10.200.0.0/16 \
  --kubeconfig=/var/lib/kube-proxy/kubeconfig \
  --proxy-mode=iptables \
  --v=2

Controller kube-scheduler flags

ExecStart=/usr/local/bin/kube-scheduler \
  --leader-elect=true \
  --master=http://127.0.0.1:8080 \
  --v=2

Controller kube-controllermanager flags

ExecStart=/usr/local/bin/kube-controller-manager \
  --address=0.0.0.0 \
  --cluster-cidr=10.200.0.0/16 \
  --cluster-name=kubernetes \
  --cluster-signing-cert-file=/var/lib/kubernetes/ca.pem \
  --cluster-signing-key-file=/var/lib/kubernetes/ca-key.pem \
  --leader-elect=true \
  --master=http://127.0.0.1:8080 \
  --root-ca-file=/var/lib/kubernetes/ca.pem \
  --service-account-private-key-file=/var/lib/kubernetes/ca-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --cloud-provider=gce \
  --allocate-node-cidrs=false \
  --configure-cloud-routes=false \
  --v=2

controller kube-apiserver flags

ExecStart=/usr/local/bin/kube-apiserver \
  --admission-control=Initializers,NamespaceLifecycle,NodeRestriction,LimitRanger,ServiceAccount,DefaultStorageClass,ResourceQuota \
  --advertise-address=10.240.0.10 \
  --allow-privileged=true \
  --apiserver-count=3 \
  --audit-log-maxage=30 \
  --audit-log-maxbackup=3 \
  --audit-log-maxsize=100 \
  --audit-log-path=/var/log/audit.log \
  --authorization-mode=Node,RBAC \
  --bind-address=0.0.0.0 \
  --client-ca-file=/var/lib/kubernetes/ca.pem \
  --enable-swagger-ui=true \
  --etcd-cafile=/var/lib/kubernetes/ca.pem \
  --etcd-certfile=/var/lib/kubernetes/kubernetes.pem \
  --etcd-keyfile=/var/lib/kubernetes/kubernetes-key.pem \
  --etcd-servers=https://10.240.0.10:2379,https://10.240.0.11:2379,https://10.240.0.12:2379 \
  --event-ttl=1h \
  --experimental-encryption-provider-config=/var/lib/kubernetes/encryption-config.yaml \
  --insecure-bind-address=127.0.0.1 \
  --kubelet-certificate-authority=/var/lib/kubernetes/ca.pem \
  --kubelet-client-certificate=/var/lib/kubernetes/kubernetes.pem \
  --kubelet-client-key=/var/lib/kubernetes/kubernetes-key.pem \
  --kubelet-https=true \
  --runtime-config=api/all \
  --service-account-key-file=/var/lib/kubernetes/ca-key.pem \
  --service-cluster-ip-range=10.32.0.0/24 \
  --service-node-port-range=30000-32767 \
  --tls-ca-file=/var/lib/kubernetes/ca.pem \
  --tls-cert-file=/var/lib/kubernetes/kubernetes.pem \
  --tls-private-key-file=/var/lib/kubernetes/kubernetes-key.pem \
  --cloud-provider=gce \
  --v=2

Versions

bofh:~$ kubectl version
Client Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:57:57Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.0", GitCommit:"6e937839ac04a38cac63e6a7a306c5d035fe7b0a", GitTreeState:"clean", BuildDate:"2017-09-28T22:46:41Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}
bofh:~$ kubectl get nodes
NAME       STATUS    ROLES     AGE       VERSION
worker-0   Ready     <none>    12d       v1.8.0
worker-1   Ready     <none>    12d       v1.8.0
worker-2   Ready     <none>    12d       v1.8.0

PV and PVC info

bofh:~$ kubectl get pv
No resources found.
bofh:~$ kubectl get pvc
No resources found.
bofh:~$ kubectl describe pv
bofh:~$ kubectl describe pvc

StorageClass config

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: standard
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-standard

Storage class output

bofh:~$ kubectl get sc
NAME       PROVISIONER
slow       kubernetes.io/gce-pd
standard   kubernetes.io/gce-pd

What about persistenceVolumes and persistenceVolumeClaims? Can you add the output of "kb describe pv" (and pvc)? — Nicola Ben
Hi @jaxxstorm. I did not have standard storageclass and I added it. I will update my config in the initial post. I still got timeouts — Johan Ryberg

jaxxstorm jaxxstorm · Accepted Answer · 2018-01-03T03:48:34

You need to add the cloud-provider flag to the apiserver, kubelet and controller-manager.

You also then need to configure a storageclass for the GCE

Kubernetes POD Timeout expired waiting for volumes to attach/mount

2 Answers