CloudFormation target group health checks are inconsistent

Question

I have an AWS CloudFormation template that creates an Application Load Balancer that routes traffic to a target group consisting of two instances running Apache.

Sometimes when I create the stack both health checks are fine as shown below:

But other times, when I create the stack using the exact same template, one or both of the health checks fail:

The portion of the template that creates the ALB and instances is:

    Type: AWS::ElasticLoadBalancingV2::LoadBalancer
    DependsOn: 
      - Ec2InstanceA
      - Ec2InstanceB
    Properties:
      IpAddressType: ipv4
      Scheme: internet-facing
      SecurityGroups:
        - !Ref InstanceSecurityGroup
      Subnets:
        - !Ref PublicSubnetA
        - !Ref PublicSubnetB
      Tags:
        - Key: Name
          Value: !Sub ${AWS::StackName}-ALB
      Type: application
  Listener:
    Type: AWS::ElasticLoadBalancingV2::Listener
    Properties:
      DefaultActions:
        - Type: forward
          TargetGroupArn: !Ref MyTargetGroup
      LoadBalancerArn: !Ref MyApplicationLoadBalancer
      Port: '80'
      Protocol: HTTP

  MyTargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties: 
      HealthCheckEnabled: true
      Port: 80
      Protocol: HTTP
      VpcId: !Ref VPC
      Targets: 
        - Id: !Ref Ec2InstanceA
        - Id: !Ref Ec2InstanceB
      TargetType: instance
  
  Ec2InstanceA:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-0323c3dd2da7fb37d
      KeyName: KeyPair
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: InstanceSecurityGroup
          SubnetId:
            Ref: PrivateSubnetA
      UserData:
        Fn::Base64:                                
          !Sub |
              #!/bin/bash -ex
              sudo yum install -y httpd;
              sudo echo "<html><h1>Hello CloudFormation A!!<h1></html>" > /var/www/html/index.html;
              cd /var/www/html;
              sudo chmod 755 index.html;
              sudo service httpd start;
              sudo chkconfig httpd on;
  Ec2InstanceB:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: t2.micro
      ImageId: ami-0323c3dd2da7fb37d
      KeyName: KeyPair
      NetworkInterfaces:
        - AssociatePublicIpAddress: true
          DeviceIndex: 0
          GroupSet:
            - Ref: InstanceSecurityGroup
          SubnetId:
            Ref: PrivateSubnetB
      UserData:
        Fn::Base64:                               
          !Sub |
              #!/bin/bash -ex
              sudo yum install -y httpd;
              sudo echo "<html><h1>Hello CloudFormation B!!<h1></html>" > /var/www/html/index.html;
              cd /var/www/html;
              sudo chmod 755 index.html;
              sudo service httpd start;
              sudo chkconfig httpd on;

I'm guessing it's some sort of resource timing issue, but I'm not really sure.

Marcin Marcin · Accepted Answer · 2020-06-20T23:06:46

Usually, when you bootstrap your instances, good practice is to use cfn-signal along with creation policy for your instances.

This ensures that the CloudFormation marks your instances as successfully created only when the UserData completes without errors. This also makes it wait until it happens.

Therefore, when ALB starts checking health of your instances, you know for sure that UserData executed successfully and your apache server is up and running.

CloudFormation target group health checks are inconsistent

2 Answers