AWS EKS Node - Volume Attach Limit

Question

We recently upgraded our EKS environment to v1.12.7. After the upgrade we noticed that there is now an "allocatable" resource called attachable-volumes-aws-ebs and in our environment we have many EBS volumes attached to each node (they were all generated dynamically via PVCs).

Yet on every node in the "allocated resources" section, it shows 0 volumes attached:

Allocatable:
 attachable-volumes-aws-ebs:  25
 cpu:                         16
 ephemeral-storage:           96625420948
 hugepages-1Gi:               0
 hugepages-2Mi:               0
 memory:                      64358968Ki
 pods:                        234
...
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests           Limits
  --------                    --------           ------
  cpu                         5430m (33%)        5200m (32%)
  memory                      19208241152 (29%)  21358Mi (33%)
  attachable-volumes-aws-ebs  0                  0

Because of this, the scheduler is continuing to try and attach new volumes to nodes that already have 25 volumes attached.

How do we get kubernetes to recognize the volumes that are attached so that the scheduler can act accordingly?

Did you find a solution to your problem? I'm running into the same issue where a node gets tainted because during a deployment EKS is trying to place more pods to a node even though the volumes are already maxed out. — Bernie Lenz
@BernieLenz Unfortunately we have not. Our company shifted focus and how it's apps were configured and this is no longer an issue for us. However, we would still love to find an answer for future reference as I'm sure it's only a matter of time before we run across it again. I bet this is something that could be answered by an AWS support engineer, but for us, the support price is way too steep. :( — viperseph

Malgorzata Malgorzata · Accepted Answer · 2019-06-19T09:02:46

First check your pods status, they can have pending status that's why probably you volumes does't count. Your volume could stuck in attaching state when using multiple PersistentVolumeClaim too.

Your volumes may be not attached due to NodeWithImpairedVolumes=true:NoSchedule flag and at the same time number of your attachable-volumes-aws-ebs.

Try to execute:

$ kubectl taint nodes node1 key:NoSchedule-

on every node with this label (NodeWithImpairedVolumes=true:NoSchedule).

If you use awsElasticBlockStore , there are some restrictions when using an awsElasticBlockStore volume:

nodes on which Pods are running must be AWS EC2 instances those instances need to be in the same region and availability-zone as the EBS volume EBS only supports a single EC2 instance mounting a volume.

You can use count/* resource quota, an object is charged against the quota if it exists in server storage. These types of quotas are useful to protect against exhaustion of storage resources use count/persistentvolumeclaims.

Allocatable will be computed by the Kubelet and reported to the API server. It is defined to be:

[Allocatable] = [Node Capacity] - [Kube-Reserved] - [System-Reserved] - [Hard-Eviction-Threshold]

Note: Since kernel usage can fluctuate and is out of kubernetes control, it will be reported as a separate value (probably via the metrics API). Reporting kernel usage is out-of-scope for this proposal.

AWS EKS Node - Volume Attach Limit

2 Answers