AWS EKS 0/1 nodes are available. 1 insufficient pods

Question

We are trying to deploy a dot net core API service to amazon EKS using ECR. The deployment was successful, but the pods are in pending status. Below are the detailed steps we followed.

Steps followed. 1. Created a docker image 2. Pushed the image to ECR. The image is now visible in aws console also. // The image looks good, I was able to run it using my docker locally.

3. Created a t2-micro cluster as below eksctl create cluster --name net-core-prod --version 1.14 --region us-west-2 --nodegroup-name standard-workers --node-type t2.micro --nodes 1 --nodes-min 1 --nodes-max 1 –managed // Cluster and Node groups were created successfully. // IAM roles also got created

4. Deployed a replication controller using the attached json/yaml//net-app.json

5. Deployed the service using the attached json/yaml //net-app-scv.json

6. The get all command returned this. //get_all.png POD always remains in PENDING status.

7. Pod describe gave the below result //describe_pod.png

8. We have also tried adding policy to the cluster IAM role to include ECR permissions attached. //ECR_policy.json

Key points:
1. We are using a t2-micro instance cluster since it’s a AWS free account.
2. We created a linux cluster and tried to push the dotnet core app. //this worked fine in our local machine
3. The cluster had only 1 node //-nodes 1 --nodes-min 1 --nodes-max 1

Can somebody please guide us on how to set up this correctly.

Answer 1

On Amazon Elastic Kubernetes Service (EKS), the maximum number of pods per node depends on the node type and ranges from 4 to 737.

If you reach the max limit, you will see something like:

❯ kubectl get node -o yaml | grep pods
      pods: "17" => this is allocatable pods that can be allocated in node
      pods: "17" => this is how many running pods you have created

If you get only one number, it should be allocatable. Another way to count all running pods is to run the following command:

kubectl get pods --all-namespaces | grep Running | wc -l

Here's the list of max pods per node type: https://github.com/awslabs/amazon-eks-ami/blob/master/files/eni-max-pods.txt

On Google Kubernetes Engine (GKE), the limit is 110 pods per node. check the following URL:

https://github.com/kubernetes/community/blob/master/sig-scalability/configs-and-limits/thresholds.md

On Azure Kubernetes Service (AKS), the default limit is 30 pods per node but it can be increased up to 250. The default maximum number of pods per node varies between kubenet and Azure CNI networking, and the method of cluster deployment. check the following URL for more information:

https://docs.microsoft.com/en-us/azure/aks/configure-azure-cni#maximum-pods-per-node

Answer 2

The issue is that you are using t2.micro. At the minimum t2.small is required. Scheduler is not able to schedule pod on the node because not enough capacity is available on the t2.micro instance. Most of the capacity is already taken by the system resources. Use t2.small at the minimum.

Answer 3

The pod was failed to be scheduled.

Issue could be with the node configuration.

share the output from below commands to identify the root cause

1.  kubectl describe node <node-name>
2.  kubectl describe pod <pod-name>
3.  kubectl get ev

There is a limit of how many pods you can run per node (regardless of resources), you're hitting that limit. You can find that limit in the output of kubectl get node -o yaml

From the node output it is clear that you've reached the limit of pods per node. Check allocated vs capacity for pods. Both are showing count has 4.

Resolution: You need to add one more node to the cluster to be able to deploy additional work load

Answer 4

Update : I tried increasing the node count to 6 and it is working now.

4 kube-system pods were already scheduled and so give it a try increasing it to 6.

I am not good at Kubes, its just a trial and error result.

It will be really useful, if someone experienced can throw some light on how this works.