Kubernetes AWS EBS Timeout waiting for volume state

4
votes

If I create this pod:

apiVersion: v1
kind: Pod
metadata:
name: dsm-manager
spec:
containers:
- name: dsm-manager
image: ******
imagePullPolicy: Always
command:
- /sbin/init
volumeMounts:
- mountPath: /srv/project/DSMManager/snapshots
name: dsm-snapshot-storage
volumes:
- name: dsm-snapshot-storage
awsElasticBlockStore:
volumeID: aws://us-west-2b/vol-43e44482
fsType: ext4
imagePullSecrets:
- name: dockerregistrykey

It always works, but if I delete it and re-create it it gets stuck with status 'CreatingContainer'. Looking in the events yields:
-Unable to mount volumes for pod "dsm-manager_default": Timeout waiting for volume state
-Error syncing pod, skipping: Timeout waiting for volume state

If I delete the pod and re-create it the same thing happens no matter what I do. However if I attach the volume to some instance and then detach it through the aws cli, then create the pod it works find. I'm wondering if the volume isn't being detached properly. For now I just have this odd work flow of attaching the volume to a random instance then detaching it while updating the container image

2
amazon-web-serviceskubernetes
Is the volume still being created when you try creating the pod that uses it? It may be caused by this issue that was just opened up today: github.com/kubernetes/kubernetes/issues/15058 – Alex Robinson
No, this is after I've already successfully created a container and mounted it. Then once I delete the pod and try to create a new one I encounter the problem. – delta440

2 Answers

6
votes

This is likely caused by a bug in the Kubernetes EBS management code, and should be fixed by PR #14493. To summarize, not validating the device block cache was causing the kubelet to think the disk was still attached after it had actually been detached.

0
votes

I've noticed a general problem with running Kubernetes and EBS volumes stuck attaching. The problem seems to be more with EBS than with Kubernetes. When the "physical" EBS volumes gets stuck in an attaching state, you can't reschedule your pods until you reboot the host. I was at a container conference recently when the team at Nordstrom who are big Kubernetes users were talking about this problem.

Your best bet is probably to use local storage on AWS for your container volumes since creating and mounting a volumes from the local Ec2 storage is not subject to the same "stuck attaching" problems as EBS. Sorry!