EKS Cluster in a private subnet - unhealthy nodes in the kubernetes cluster

Question

I'm trying to create a EKS cluster in a private subnet. I'm having issues getting it working. I get the error unhealthy nodes in the kubernetes cluster. Wonder if its due to security group or some other issues like VPC endpoints?

When I use NAT gateway setup then it works fine. But I don't want to use nat gateway anymore.

One think I'm not sure is should the EKS cluster subnet_ids be only private subnets?

In the below config I'm using both public and private subnets.

resource "aws_eks_cluster" "main" {
  name      = var.eks_cluster_name
  role_arn  = aws_iam_role.eks_cluster.arn

  vpc_config {
    subnet_ids              = concat(var.public_subnet_ids, var.private_subnet_ids)
    security_group_ids      = [aws_security_group.eks_cluster.id, aws_security_group.eks_nodes.id, aws_security_group.external_access.id]
    endpoint_private_access = true
    endpoint_public_access  = false
  }

  # Ensure that IAM Role permissions are created before and deleted after EKS Cluster handling.
  # Otherwise, EKS will not be able to properly delete EKS managed EC2 infrastructure such as Security Groups.

  depends_on = [
    "aws_iam_role_policy_attachment.aws_eks_cluster_policy",
    "aws_iam_role_policy_attachment.aws_eks_service_policy"
  ]
}

Do you have any NAT gateway to provide internet access for nodes in private subnets? — Marcin
No nat gateway. I don't want to use Nat gateway since its price is higher. I had a different setup which had nat gateway and that worked fine. But I want to implement without a nat gateway and use vpc endpoints. — John Doe
What should be the method for using VPC endpoints instead of nat gateway? — John Doe

Marcin Marcin · Accepted Answer · 2021-01-06T09:29:26

Since you don't have NAT gateway/instance, your nodes can't connect to the internet and fail as they can't "communicate with the control plane and other AWS services" (from here).

Thus, you can use VPC endpoints to enable communication with the plain and the services. To view the properly setup VPC with private subnets for EKS, you can check AWS provided VPC template for EKS (from here).

From the template, the VPC endpoints in us-east-1:

com.amazonaws.us-east-1.ec2
com.amazonaws.us-east-1.ecr.api
com.amazonaws.us-east-1.s3
com.amazonaws.us-east-1.logs
com.amazonaws.us-east-1.ecr.dkr
com.amazonaws.us-east-1.sts

Please note that all these endpoints, escept S3, are not free. So you have to consider if running cheap NAT instances or gateway would be cheaper or more expensive then maintaining these endpoints.

EKS Cluster in a private subnet - unhealthy nodes in the kubernetes cluster

1 Answers