aws auto scaling remove unhealthy instance before it scales up rendering the application unusable

Question

I am testing out aws auto scaling and other corresponding services.

I have 1 instance registered to target group

minimum/desired instances on auto scaling group as 1. Scale up when CPU utilisation above 50%

During load testing, due to increase in server response time, aws target group fails health check and starts draining the 1 instance that is serving behind my aws application load balancer. Afterwards auto scaling group starts removing the unhealthy instance and my web service goes down.

What should happen is that aws auto scaling group should spin off new insurance as soon as there is an increase in load on the server.

If I increase the minimum/desired server to 2 or 4, same results, on load testing, all servers receive heavy traffic, target group health check fails and it marks all of them unhealthy and aws asg starts removing them all before detecting high CPU utilisation and spawn new instances.

What should be done here?

Answer 1

The speed in which your instances are added to the autoscaling group is based on the alarm setup in the execution policy.

The value has to be reach the alarm conditions before a new host is added, after this the autoscaling groups grace period will grant a duration of time before instance health is evaluated as well as before evaluations for the alarm can happen again.

If you want to scale up quicker you should look at adjusting these values, in addition if you need to scale to use variable amounts of instances take a look at step scaling over simple scaling.

Finally during load testing try need to ramp up over time rather than immediately to represent more natural traffic. This will your application time to naturally scale.