Difficulty using Kubernetes Persistent Volume Claims with Amazon EBS

Question

I'm attempting to follow the instructions at https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/registry to add a private docker registry to Kubernetes, but the pod created by the rc isn't able to mount the persistent volume claim.

First I'm creating a volume on EBS like so:

aws ec2 create-volume --region us-west-1 --availability-zone us-west-1a --size 32 --volume-type gp2

(us-west-1a is also the availability zone that all of my kube minions are running in.)

Then I create a persistent volume like so:

kind: PersistentVolume
apiVersion: v1
metadata:
  name: kube-system-kube-registry-pv
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  awsElasticBlockStore:
    volumeID: vol-XXXXXXXX
    fsType: ext4

And a claim on the persistent volume like so:

kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: kube-registry-pvc
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 30Gi

The replication controller is specified like so:

apiVersion: v1
kind: ReplicationController
metadata:
  name: kube-registry-v0
  namespace: kube-system
  labels:
    k8s-app: kube-registry
    version: v0
    kubernetes.io/cluster-service: "true"
spec:
  replicas: 1
  selector:
    k8s-app: kube-registry
    version: v0
  template:
    metadata:
      labels:
        k8s-app: kube-registry
        version: v0
        kubernetes.io/cluster-service: "true"
    spec:
      containers:
      - name: registry
        image: registry:2
        resources:
          limits:
            cpu: 100m
            memory: 100Mi
        env:
        - name: REGISTRY_HTTP_ADDR
          value: :5000
        - name: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY
          value: /var/lib/registry
        volumeMounts:
        - name: image-store
          mountPath: /var/lib/registry
        ports:
        - containerPort: 5000
          name: registry
          protocol: TCP
      volumes:
      - name: image-store
        persistentVolumeClaim:
          claimName: kube-registry-pvc

When I create the rc, It successfully starts a pod, but the pod is unable to mount the volume:

$ kubectl describe po kube-registry --namespace=kube-system
...
Events:
  FirstSeen     LastSeen        Count   From                                                    SubobjectPath   Reason          Message
  ─────────     ────────        ─────   ────                                                    ─────────────   ──────          ───────
  1m            1m              1       {scheduler }                                                            Scheduled       Successfully assigned kube-registry-v0-3jobf to XXXXXXXXXXXXXXX.us-west-1.compute.internal
  22s           22s             1       {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal}                    FailedMount     Unable to mount volumes for pod "kube-registry-v0-3jobf_kube-system": Timeout waiting for volume state
  22s           22s             1       {kubelet XXXXXXXXXXXXXXX.us-west-1.compute.internal}                    FailedSync      Error syncing pod, skipping: Timeout waiting for volume state

I'm able to successfully mount EBS volumes if I don't use persistent volumes and persistent volume claims. The following works without error, for example:

apiVersion: v1
kind: Pod
metadata:
  name: test-ebs
spec:
  containers:
  - image: gcr.io/google_containers/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /test-ebs
      name: test-volume
  volumes:
  - name: test-volume
    awsElasticBlockStore:
      volumeID: vol-XXXXXXXX
      fsType: ext4

My two questions are:

Does anyone know what might be going wrong and how to fix it?
In general, where can I look for more details on errors like these? I haven't been able to find more detailed log messages anywhere, and "Unable to mount volumes...Timeout waiting for volume state" isn't terribly helpful.

Did you check the Kubernetes logs on the node where the pod was started? Your answer should be there. — Antoine Cotten
I went through the same exercise and noticed that I can skip the pv and use dynamic pv provisioning instead by defining a storageclass with aws-ebs as the provisioner. And I managed to complete the registry setup doing so. As for trouble-shooting, use "describe". I did encounter a pvc config bug in first try becasue aws sc1 storage requires a minimum of 500Gi and I spec'd something smaller. — bonchef

mlucy mlucy · Accepted Answer · 2016-02-10T02:49:40

I think I was likely running into https://github.com/kubernetes/kubernetes/issues/15073 . (If I create a new EBS volume, I first get a different failure, and then after the pod has been killed if I try to re-create the rc I get the failure I mentioned in my question.)

Also, for anyone else wondering where to look for logs, /var/log/syslog and /var/log/containers/XXX on the kubelet was where I ended up having to look.

Difficulty using Kubernetes Persistent Volume Claims with Amazon EBS

1 Answers