How to prevent Google Cloud Load balancer to forward the traffic to newly created auto scaled Instance without being ready?

Question

I will need to host a PHP Laravel application on Google Cloud Compute Engine with auto scaling and load balancing. I tried to setup and configure following:

1. I Created instance template, where I have added startup script to install apache2, PHP, cloning the git repository of my project, Configuring the Cloud SQL proxy, and configure all settings required to run this Laravel project.

2. Created Instance group, Where I have configured a rule when CPU reaches certain percent it start creating other instances for auto scale.

3. Created Cloud SQL instance.

4. Created Storage bucket, in my application all of the public contents like images will be uploaded into storage bucket and it will be served from there.

5. Created Load Balancer and assigned the Public IP to load balancer, configured the fronted and backed correctly for load balancer.

As per my above configuration, everything working fine, When a instance reaches a defined CPU percentage, Auto scaling start creating another instances and load balancer start routing the traffic to new instance.

The issue I'm getting, to configure and setup my environment(the startup script of instance template) takes about 20-30 minutes to configure and start ready to serve the content from the newly created instance. But when the load balancer detects if the newly created machine is UP and running it start routing the traffic to new VM instance which is not being ready to serve the content from it.

As a result, when load balancer routes the traffic to not ready machine, it obviously send me 404 error, and some other errors.

How to prevent to happen it, is there any way that the instance that created through auto scaling service send some information to load balancer after this machine is ready to serve the content and then only the load balancer route the traffic to the newly created instance?

Answer 1

How to prevent Google Cloud Load balancer to forward the traffic to newly created auto scaled Instance without being ready?

Google Load Balancers use the parameter Cool Down to determine how long to wait for a new instance to come online and be 100% available. However, this means that if your instance is not available at that time, errors will be returned.

The above answers your question. However, taking 20 or 30 minutes for a new instance to come online defeats a lot of the benefits of autoscaling. You want instances to come online immediately.

Best practices mean that you should create an instance. Configure the instance with all the required software applications, etc. Then create an image of this instance. Then in your template specify this image as your baseline image. Now your instances will not have to wait for software downloads and installs, configuration, etc. All you need to do is run a script that does the final configuration, if needed, to bring an instance online. Your goal should be 30 - 180 seconds from launch to being online and running for a new instance. Rethink / redesign anything that takes longer than 180 seconds. This will also save you money.

Answer 2

John Hanley answer is pretty good, I'm just completing it a bit.

You should take a look at packer to create your preconfigured google images, this will help you when you need to add a new configuration or do updates.

The cooldown is a great way, but in your case you can't really be sure that your installation won't take a bit more time sometimes due to updates as you should do an apt-get update && apt-get upgrade at instance startup to be up to date it will only take more and more time...

Load balancers normally should have a health check configured and should not route traffic unless the instance is detected as healthy. In your case as you have apache2 installed I suppose you have a HC on the port 80 or 443 depending on your configuration on a /healthz path.

A way to use the health check correctly would be to create a specific vhost for the health check and you add a fake domain in the HC, let's say health.test, that would give a vhost listening for health.test and returning a 200 response on /healthz path. This way if you don't change you conf, just activate the health vhost last so the loadbalancer don't start routing traffic before the server is really up...