AWS autoscaling group health check type : ELB or EC2?

3
votes

I am creating an internet-facing classic load balancer (CLB) with configurations as below

Connection Draining:Enabled, 300 seconds

health check configuration 
Ping Target 
TCP:22
Timeout 5 seconds
Interval    30 seconds
Unhealthy threshold 2
Healthy threshold   10

and attach this CLB with an auto scaling group (ASG) with below configurations

Desired: 2
Availability Zone(s) : us-east-1a, us-east-1c
Min:2
Subnet(s) : subnet-0ab1df27,subnet-5dc8af06
Max: 20
Health Check Type: ELB
Health Check Grace Period: 300
Default Cooldown:300

below are the scaling policy attached with ASG

increase-instance-policy

Execute policy when: aws-ec2-alarm-CPU-high-utilization
breaches the alarm threshold: CPUUtilization >= 70 for 2 consecutive periods of 300 seconds for the metric dimensionsAutoScalingGroupName = demo-auto-scaling-group
Take the action: Add 1 instances
And then wait 300 seconds before allowing another scaling activity

decrease-instance-policy

Execute policy when: aws-ec2-alarm-CPU-low-utilization
breaches the alarm threshold: CPUUtilization <= 30 for 2 consecutive periods of 300 seconds the metric dimensions = demo-auto-scaling-group
Take the action: Remove 1 instances
And then wait 300seconds before allowing another scaling activity

BUT with above configurations instances are keep creating and terminating simultaneously without any CPU utilization?

Here is the activity history tab of ASG

Description:DescriptionLaunching a new EC2 instance: i-0768deac4d37591eb Cause:CauseAt 2017-01-03T10:11:42Z an instance was started in response to a difference between desired and actual capacity, increasing the capacity from 1 to 2.

Description:DescriptionTerminating EC2 instance: i-0768deac4d37591eb Cause:CauseAt 2017-01-03T10:19:02Z an instance was taken out of service in response to a ELB system health check failure.

also on ELB page, instance tab list the instances but the status are always shown OutOfService with the info

"Instance has failed at least the UnhealthyThreshold number of health checks consecutively."

But if I change Health Check Type: from ELB to EC2 than everything works good; no multiple instances creations/termination

but Here in this article I read that ELB health check is better than EC2 as it check EC instance health underneath;

--

kindly suggest me what wrong I am doing here; may be some time issue? as I am creating 2 instances and default cooldown time is 300 second and and meanwhile alarm is being is alarmed or any other issue?

Thanks

2
amazon-web-servicesamazon-ec2aws-sdkautoscaling
Do you manually set the desired value? – Mahdi
no. all I did through AWS SDK – diEcho
In your configuration I see you set the desired value to 2. It does not seem correct to me. Because the desired value is for forcing the ASG to change the number of instances regardless of the alarm. normally, it should not be set. – Mahdi
yes. I always want atleast 2 instance to running ; from AWS SDK "DesiredCapacity — (Integer) The number of EC2 instances that should be running in the Auto Scaling group. This number must be greater than or equal to the minimum size of the group and less than or equal to the maximum size of the group." – diEcho
but the termination happened because of ELB health check failure. isn't it? – diEcho

2 Answers

3
votes

When using ELB health check, the Auto Scaling Group will rely on the ELB health check to determine if an instance is healthy or not. It looks like the ELB health check is failing, which causes the ASG to terminate the instance and launch a new one.

Are you sure the ELB health check is correctly configured? It looks you are checking port 22. The ELB will need to be able to access this port. Is your security group allowing this?

2
votes

The issue is in the configuration of your load balancer. Change it as follows:

Timeout 30 seconds
Interval 60 seconds
Unhealthy threshold 10
Healthy threshold 2