I want to setup AWS EKS cluster (AWS::EKS::Cluster) and worker nodes group (AWS::AutoScaling::AutoScalingGroup) in a single CloudFormation stack. This is the CF definition I created:
AWSTemplateFormatVersion: "2010-09-09"
Description: Creates API gateway and services for my projects
Parameters:
ClusterName:
Type: String
Description: Cluster name
Default: eks-min-cluster
NodeAutoScalingGroupDesiredCapacity:
Type: Number
Default: 2
Description: Desired capacity of Node Group ASG.
NodeAutoScalingGroupMinSize:
Type: Number
Default: 1
Description: Minimum size of Node Group ASG.
NodeAutoScalingGroupMaxSize:
Type: Number
Default: 3
Description: Maximum size of Node Group ASG. Set to at least 1 greater than NodeAutoScalingGroupDesiredCapacity.
BootstrapArguments:
Type: String
Default: ""
Description: "Arguments to pass to the nodes' bootstrap script. See files/bootstrap.sh in https://github.com/awslabs/amazon-eks-ami"
VpcCidr:
Description: Please enter the IP range (CIDR notation) for this VPC
Type: String
Default: 10.192.0.0/16
PublicSubnet1Cidr:
Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
Type: String
Default: 10.192.20.0/24
PublicSubnet2Cidr:
Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone
Type: String
Default: 10.192.21.0/24
PrivateSubnet1Cidr:
Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone
Type: String
Default: 10.192.22.0/24
PrivateSubnet2Cidr:
Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone
Type: String
Default: 10.192.23.0/24
NodeImageIdSSMParam:
Type: "AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>"
Default: /aws/service/eks/optimized-ami/1.14/amazon-linux-2/recommended/image_id
Description: AWS Systems Manager Parameter Store parameter of the AMI ID for the worker node instances.
Resources:
InternetGateway:
Type: AWS::EC2::InternetGateway
Vpc:
Type: AWS::EC2::VPC
Properties:
CidrBlock: !Ref VpcCidr
EnableDnsSupport: true
EnableDnsHostnames: true
VpcGatewayAttachment:
Type: AWS::EC2::VPCGatewayAttachment
Properties:
InternetGatewayId: !Ref InternetGateway
VpcId: !Ref Vpc
PublicSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: !Ref PublicSubnet1Cidr
MapPublicIpOnLaunch: true
PublicSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: !Ref PublicSubnet2Cidr
MapPublicIpOnLaunch: true
PrivateSubnet1:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
AvailabilityZone: !Select [ 0, !GetAZs '' ]
CidrBlock: !Ref PrivateSubnet1Cidr
MapPublicIpOnLaunch: true
PrivateSubnet2:
Type: AWS::EC2::Subnet
Properties:
VpcId: !Ref Vpc
AvailabilityZone: !Select [ 1, !GetAZs '' ]
CidrBlock: !Ref PrivateSubnet2Cidr
MapPublicIpOnLaunch: true
SshSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
VpcId: !Ref Vpc
GroupDescription: Enable SSH access via port 22
SecurityGroupIngress:
- CidrIp: 0.0.0.0/0
FromPort: 22
IpProtocol: tcp
ToPort: 22
- CidrIp: 0.0.0.0/0
FromPort: 8
IpProtocol: icmp
ToPort: -1
GatewayHostSshPortAddress:
Type: AWS::EC2::EIP
DependsOn: VpcGatewayAttachment
Properties:
Domain: vpc
AssociateGatewayHostSshPort:
Type: AWS::EC2::EIPAssociation
DependsOn: GatewayHostSshPortAddress
Properties:
AllocationId: !GetAtt GatewayHostSshPortAddress.AllocationId
NetworkInterfaceId: !Ref GatewayHostSshNetworkInterface
GatewayHostSshNetworkInterface:
Type: AWS::EC2::NetworkInterface
Properties:
SubnetId: !Ref PublicSubnet1
Description: Interface for controlling traffic such as SSH
GroupSet:
- !Ref SshSecurityGroup
SourceDestCheck: true
PublicRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
PrivateSubnet1RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
PrivateSubnet2RouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
DefaultPublicRoute:
Type: AWS::EC2::Route
DependsOn: VpcGatewayAttachment
Properties:
RouteTableId: !Ref PublicRouteTable
DestinationCidrBlock: 0.0.0.0/0
GatewayId: !Ref InternetGateway
PrivateSubnet1Route:
Type: AWS::EC2::Route
DependsOn:
- VpcGatewayAttachment
- PrivateSubnet1NatGateway
Properties:
RouteTableId: !Ref PrivateSubnet1RouteTable
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref PrivateSubnet1NatGateway
PrivateSubnet2Route:
Type: AWS::EC2::Route
DependsOn:
- VpcGatewayAttachment
- PrivateSubnet2NatGateway
Properties:
RouteTableId: !Ref PrivateSubnet2RouteTable
DestinationCidrBlock: 0.0.0.0/0
NatGatewayId: !Ref PrivateSubnet2NatGateway
PrivateSubnet1NatGateway:
Type: AWS::EC2::NatGateway
DependsOn:
- PrivateSubnet1NatGatewayEIP
- PublicSubnet1
- VpcGatewayAttachment
Properties:
AllocationId: !GetAtt PrivateSubnet1NatGatewayEIP.AllocationId
SubnetId: !Ref PublicSubnet1
PrivateSubnet2NatGateway:
Type: AWS::EC2::NatGateway
DependsOn:
- PrivateSubnet2NatGatewayEIP
- PublicSubnet2
- VpcGatewayAttachment
Properties:
AllocationId: !GetAtt PrivateSubnet2NatGatewayEIP.AllocationId
SubnetId: !Ref PublicSubnet2
PrivateSubnet1NatGatewayEIP:
DependsOn:
- VpcGatewayAttachment
Type: 'AWS::EC2::EIP'
Properties:
Domain: vpc
PrivateSubnet2NatGatewayEIP:
DependsOn:
- VpcGatewayAttachment
Type: 'AWS::EC2::EIP'
Properties:
Domain: vpc
PublicRouteTableToPublicSubnet1Association:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet1
PublicRouteTableToPublicSubnet2Association:
Type: AWS::EC2::SubnetRouteTableAssociation
Properties:
RouteTableId: !Ref PublicRouteTable
SubnetId: !Ref PublicSubnet2
PrivateRouteTable:
Type: AWS::EC2::RouteTable
Properties:
VpcId: !Ref Vpc
GatewayHost:
Type: AWS::EC2::Instance
DependsOn: [AssociateGatewayHostSshPort]
Properties:
ImageId: ami-03c3a7e4263fd998c
InstanceType: t2.nano
AvailabilityZone: !Select [ 0, !GetAZs '' ]
KeyName: jd-system
NetworkInterfaces:
-
NetworkInterfaceId: !Ref GatewayHostSshNetworkInterface
DeviceIndex: 0
Metadata:
AWS::CloudFormation::Init:
config:
files:
/etc/kong/kong.yml:
content: test-jd
#source:
mode: "000644"
owner: "root"
group: "root"
EksIamRole:
Type: 'AWS::IAM::Role'
Properties:
AssumeRolePolicyDocument:
Version: 2012-10-17
Statement:
- Effect: Allow
Principal:
Service:
- eks.amazonaws.com
Action:
- 'sts:AssumeRole'
RoleName: EksIamRole
ManagedPolicyArns:
- arn:aws:iam::aws:policy/AmazonEKSClusterPolicy
- arn:aws:iam::aws:policy/AmazonEKSServicePolicy
################### CONTROL PLANE ###################
ClusterControlPlaneSecurityGroup:
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: Cluster communication with worker nodes
VpcId: !Ref Vpc
EksCluster:
Type: AWS::EKS::Cluster
Properties:
Name: !Ref ClusterName
RoleArn: !GetAtt EksIamRole.Arn
ResourcesVpcConfig:
SecurityGroupIds:
- !Ref SshSecurityGroup
- !Ref ClusterControlPlaneSecurityGroup
SubnetIds:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
DependsOn: [EksIamRole, PublicSubnet1, PublicSubnet2, PrivateSubnet1, PrivateSubnet2, SshSecurityGroup]
################### WORKER NODES ###################
NodeSecurityGroup:
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: Security group for all nodes in the cluster
Tags:
- Key: !Sub kubernetes.io/cluster/${ClusterName}
Value: owned
VpcId: !Ref Vpc
NodeSecurityGroupIngress:
Type: "AWS::EC2::SecurityGroupIngress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow node to communicate with each other
FromPort: 0
GroupId: !Ref NodeSecurityGroup
IpProtocol: "-1"
SourceSecurityGroupId: !Ref NodeSecurityGroup
ToPort: 65535
ClusterControlPlaneSecurityGroupIngress:
Type: "AWS::EC2::SecurityGroupIngress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow pods to communicate with the cluster API Server
FromPort: 443
GroupId: !Ref ClusterControlPlaneSecurityGroup
IpProtocol: tcp
SourceSecurityGroupId: !Ref NodeSecurityGroup
ToPort: 443
ControlPlaneEgressToNodeSecurityGroup:
Type: "AWS::EC2::SecurityGroupEgress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow the cluster control plane to communicate with worker Kubelet and pods
DestinationSecurityGroupId: !Ref NodeSecurityGroup
FromPort: 1025
GroupId: !Ref ClusterControlPlaneSecurityGroup
IpProtocol: tcp
ToPort: 65535
ControlPlaneEgressToNodeSecurityGroupOn443:
Type: "AWS::EC2::SecurityGroupEgress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow the cluster control plane to communicate with pods running extension API servers on port 443
DestinationSecurityGroupId: !Ref NodeSecurityGroup
FromPort: 443
GroupId: !Ref ClusterControlPlaneSecurityGroup
IpProtocol: tcp
ToPort: 443
NodeSecurityGroupFromControlPlaneIngress:
Type: "AWS::EC2::SecurityGroupIngress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow worker Kubelets and pods to receive communication from the cluster control plane
FromPort: 1025
GroupId: !Ref NodeSecurityGroup
IpProtocol: tcp
SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
ToPort: 65535
NodeSecurityGroupFromControlPlaneOn443Ingress:
Type: "AWS::EC2::SecurityGroupIngress"
DependsOn: NodeSecurityGroup
Properties:
Description: Allow pods running extension API servers on port 443 to receive communication from cluster control plane
FromPort: 443
GroupId: !Ref NodeSecurityGroup
IpProtocol: tcp
SourceSecurityGroupId: !Ref ClusterControlPlaneSecurityGroup
ToPort: 443
NodeInstanceRole:
Type: "AWS::IAM::Role"
Properties:
AssumeRolePolicyDocument:
Version: "2012-10-17"
Statement:
- Effect: Allow
Principal:
Service:
- ec2.amazonaws.com
Action:
- "sts:AssumeRole"
ManagedPolicyArns:
- "arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy"
- "arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy"
- "arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly"
Path: /
NodeInstanceProfile:
Type: "AWS::IAM::InstanceProfile"
Properties:
Path: /
Roles:
- Ref: NodeInstanceRole
NodeLaunchConfig:
Type: "AWS::AutoScaling::LaunchConfiguration"
Properties:
AssociatePublicIpAddress: "true"
BlockDeviceMappings:
- DeviceName: /dev/xvda
Ebs:
DeleteOnTermination: true
VolumeSize: 10
VolumeType: gp2
IamInstanceProfile: !Ref NodeInstanceProfile
#ImageId: ami-03c3a7e4263fd998c
ImageId: !Ref NodeImageIdSSMParam
InstanceType: t2.nano
KeyName: jd-system
SecurityGroups:
- Ref: NodeSecurityGroup
UserData: !Base64
"Fn::Sub": |
#!/bin/bash
set -o xtrace
/etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
/opt/aws/bin/cfn-signal --exit-code $? \
--stack ${AWS::StackName} \
--resource NodeGroup \
--region ${AWS::Region}
NodeGroup:
Type: "AWS::AutoScaling::AutoScalingGroup"
DependsOn:
- EksCluster
- Vpc
Properties:
DesiredCapacity: !Ref NodeAutoScalingGroupDesiredCapacity
LaunchConfigurationName: !Ref NodeLaunchConfig
MaxSize: !Ref NodeAutoScalingGroupMaxSize
MinSize: !Ref NodeAutoScalingGroupMinSize
Tags:
- Key: Name
PropagateAtLaunch: "true"
Value: !Sub ${ClusterName}-NodeGroup-Node
- Key: !Sub kubernetes.io/cluster/${ClusterName}
PropagateAtLaunch: "true"
Value: owned
VPCZoneIdentifier:
- !Ref PublicSubnet1
- !Ref PublicSubnet2
- !Ref PrivateSubnet1
- !Ref PrivateSubnet2
UpdatePolicy:
AutoScalingRollingUpdate:
MaxBatchSize: "1"
MinInstancesInService: !Ref NodeAutoScalingGroupDesiredCapacity
PauseTime: PT5M
Outputs:
GatewayHostPublicIp:
Description: Gateway host public ip
Value: !GetAtt GatewayHost.PublicIp
EksClusterEndpoint:
Description: EksCluster endpoint
Value: !GetAtt EksCluster.Endpoint
After stack creation I cannot see any worker nodes:
$ kubectl get nodes
No resources found
Nor pods are getting created:
$ kubectl get pods --all-namespaces
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-59b69b4849-l97bq 0/1 Pending 0 7m15s
kube-system coredns-59b69b4849-zwtql 0/1 Pending 0 7m15s
kube-system metrics-server-7949d47784-2xjck 0/1 Pending 0 8s
Tutorials I read create EKS cluster via one CF stack and worker nodes group via another. I want to setup everything via single script. I suspect that the worker nodes group is getting created too quickly, however, I am new to CF and EKS and cannot confirm that. Please advise.
