AWS Application load balancer - wait some time before executing a healthcheck

Question

I have an Application load balancer that is configured to forward requests to EC2 instances inside a target group. Also, I have an autoscaling group attached to the same target group which is increasing the number of instances based on the server load.

Healthchecks are configured inside the target group and are working okay. However, sometimes when the load is increased and the autoscaling group creates a new instance, I see an exception being raised on the new EC2 instance when the healthcheck is invoked by the ELB.

Usually, my instances need around 30 seconds to install some libraries after they are spinned up. That's why I have quite a generous Health Check Grace Period configured inside the autoscaling group.

The problem, that is bugging me is that it seems the load balancer is sometimes requesting the healthcheck endpoint (even though I have configured 5 mins Interval inside the target group) before the instance has been installed the needed libraries. This causes an exception to be thrown and I receive a notification for that, which is very annoying.

It seems that Health Check Grace Period inside Auto Scaling does not stop the ELB from sending healthcheck requests to the new instance. Do you have an idea how to make the ELB healthcheck to wait some time after autoscaling creates a new instance?

Answer 1

I ended up here via. Google, and was sad to not see an answer. I've later found a resource that makes this work via Lifecycle hooks (found under instance management in the auto scaling group). With the lifecycle hook, you can basically "pause" the instance in the pre-inService state, until either the pause is canceled via. a commandline call, or a timeout occurs.

So in this case, you'd add something like this to the startup script/user data:

INSTANCE_ID="`wget -q -O - http://instance-data/latest/meta-data/instance-id`" && \
aws autoscaling complete-lifecycle-action --lifecycle-action-result CONTINUE --instance-id $INSTANCE_ID --lifecycle-hook-name test-hook --auto-scaling-group-name test-asg --region eu-west-1 

Make sure to install the aws cli during other installations on the server, and change the lifecycle name, group name and region to fit.

More information can be found here: https://medium.com/faun/autoscaling-like-a-pro-how-to-use-ec2-auto-scaling-lifecycle-hooks-the-right-way-7e1e6c952d3c