AWS ELB Auto scaling on CPU Utilization > 90% - constantly creating / terminating instances

Question

I have an auto scale group with triggers as follows:

Average CPU Utliziation > 90% scale up 1 instance
Average CPU Utilization < 25% scale down 1 instance

The metric is being calculated every 2 minutes and the breach limit is 10 minutes.

The problem I am experiencing is that the triggers are being triggered constantly it seems. The instances are being created and destroyed every 10 minutes. I have been monitoring the CPU Utilization and it never surpasses the scale up threshold. The maximium it hits is around 80% and this only happened 1 time, most of the time it is in the 20 to 25% range. I only have 1 instance running normally, but eveyr 10 minutes ELB will create a new instance, and soon after it will terminate it.

Any thing I am doing wrong here? Am I not understanding how the average CPU Utilization works?

Answer 1

The new EC2 instances are being created by Auto-Scaling (not Load Balancer).

There is a "Scaling History" tab in the Auto Scaling group that might provide some hints as to what is triggering the scale-out policy.

Check whether "Detailed Monitoring" is enabled on the Auto Scaling group and/or Launch Configuration -- this will cause metrics (eg CPU) to be collected every 1 minute instead of the default 5 minutes.

Check the setting on your CloudWatch chart to match the metric collection interval -- if metrics are being collected every minute, set the CloudWatch chart to 1-minute also. Otherwise, you might be viewing metrics at a lower "resolution" than the alarm itself.

Worst case, increase the timing settings for the Alarm, such as "Above 90% for 2 consecutive periods" rather than just one period.