Cloud formation - updating a stack behind an elb doesnt update the AMI

6
votes

We are powering our AWS EC2 instances using cloud formation. We have 3 different stacks - testing, staging and production. Our workflow to update the images for any of the stacks is as follows:

  1. Update a 'golden master' instance
  2. Snapshot the golden master to a disk image
  3. Change the ami reference in our cloud-formation config (via a json file) for a given stack and update the stack.

This brings down the instances in the stack + re-provisions them with the new disk image.

We've had no problems with our testing or staging stacks, which contain a single ec2 instance each. Each time we update, the image is replaced no problem.

Our production stack doesn't seem to be working in the same way :-(. It contains (at least) 2 instances sitting behind a load balancer. When we update this stack in the same way, the ec2 instances are not refreshed straight away (i.e after the update is completed, the boxes are still running from the previous disk image). The good news is the new images are used when the load balancer autoscales.

Could there be a conflict between the load balancing rules and cloud formation?

Any insight would be greatly appreciated

2
amazon-web-servicesamazon-ec2autoscalingamazon-elbamazon-cloudformation
Can you include your Cloud Formation script? Also - have you got an UpdatePolicy defined? – Pete - MSFT

2 Answers

7
votes

OK - Done some experiments. You need to define an update policy for the scaling group. By defining an update policy, any change in the launch configuration will then trigger an update. Without this - an update won't be triggered.

You don't need an ELB in order to trigger the rolling update, but if you have one, it also won't interfere with it.

http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html

0
votes

May we see your cloud formation script? You are probably right concerning elb + autoscaling.

In order to update ami (without cloudformation) I double the desiredCapacity value of the autoscaling group manually and set it back once the new instances are status online at the ELB.

A similar tactics might be scriptable/configurable with cloudformation