CloudFormation, CodeDeploy, ELB & Auto-Scaling Group

Question

I am trying to build a stack with an ELB, an Auto-Scaling Group and a Pipeline (with CodeBuild and CodeDeploy). I can't understand how it is supposed to work: the auto-scaling group is starting two instances and wait X minutes before starting to check the instances state the CodeDeploy application deployment group is waiting for the Auto-Scaling group to be created and ready the pipeline takes about 10 minutes to start deploying the application

My issue is when I create the stack, it looks like there is a loop: AG requires an application from CodeDeploy and CodeDeploy requires an AG stabilized. To be clear, when the application is ready to deploy, my Auto-Scaling group is already starting to terminate instances and starting new ones, so the CodeDeployment is trying to deploy to instances already terminated or terminating.

I don't really want to configure HealthCheckGracePeriod and PauseTime to be ~10-15 minutes... it is way too long.

Are there any best practices for CloudFormation + ELB + AG + CodeDeploy via a Pipeline? What should be the steps to achieve that?

Thank you!

Answer 1

This stopping/staring the instances is most probably linked to the Deployment Type: in-place vs. blue/green. I have tried both in my setup, and I will try to summarize how they work.

Let's say that for this example, you have an Autoscaling group which at the time of deploying the application has 2 running instances and the deployment configuration is OneAtATime. Traffic is controlled by the Elastic Load Balancer. Then:

In-place deployment:

1. CodeDeploy gets notified of a new revision available.
2. It tells the ELB to stop directing traffic to 1st instance.
3. Once traffic to one instance is stopped, it starts the deployment process: Stop the application, download bundle etc.
4. If the deployment is successful (validate service hook returned 0), it tells ELB to resume traffic to that instance.
5. At this point, 1 instance is running the old code and 1 is running the new code.
6. Right after the ELB stops traffic to the 2nd instance, and repeats the deployment process there.

Important note: With ELB enabled, the time it takes to block traffic to instance before deployment, and time it takes to allow traffic after it are directly dependent on your health check: time = Healthy threshold * Interval.

Blue/green deployment:

1. CodeDeploy gets notified of a new revision available.
2. It copies your Autoscaling Group: the same configuration of the group (including scaling policies, scheduled actions etc.) and the same number of instances (using same AMI as your original AG) there were there at the start of deployment - in our case 2.
3. At this point, there is no traffic going to the new AG.
4. CodeDeploy performs all the usual installation steps to one machine.
5. If successful, it deploys to the second machine too.
6. It directs traffic from the instances in your old AG to the new AG.
7. Once traffic is completely re-routed, it deletes the old AG and terminates all its instances (after a period specified in Deployment Settings - this option is only available if you select Blue/Green)
8. Now ELB is serving only the new AG.

From experience:

• Blue/green deployment is a bit slower, since you need to wait for the machines to boot up, but you get a much safer and fail-proof deployment.
• In general I would stick with Blue/green, with load balancer enabled and the Deployment Configuration: AllAtOnce (if it fails, customers won't be affected since the instances won't be receiving traffic. But it will be 2x as fast since you deploy in parallel rather than sequentially).
• If your health checks and validate service are throughout enough, you can probably delete the original AG with minimal waiting time (5 minutes at the time of writing the post).