We configure our Launch Configuration to use a "clean" off-the-shelf AMI - we use these: http://aws.amazon.com/amazon-linux-ami/
One of the features of these AMIs is CloudInit - https://help.ubuntu.com/community/CloudInit
This feature enables us to deliver to the newly spawned plain vanilla EC2 instance some data. Specifically, we give the instance a script to run.
The script (in a nutshell) does the following:
- Upgrades itself (to make sure all security patches and bug fixes are applied).
- Installs Git and Puppet.
- Clones a Git repo from Github.
- Applies a puppet script (which is part of the repo) to configure itself. Puppet installs the rest of the needed software modules.
It does take longer than booting from a pre-configured AMI, but we skip the process of actually making these AMIs every time we update the software (a couple of times a week) and the servers are always "clean" - no manual patches, all software modules are up to date etc.
Now, to upgrade the software, we use a local boto script.
The script kills the servers running the old code one by one. The Auto Scaling mechanism launches new (and upgraded) servers.
Make sure to use as-terminate-instance-in-auto-scaling-group
because using ec2-terminate-instance
will cause the ELB to continue to send traffic to the shutting-down instance, until it fails the health check.
Interesting related blog post: http://blog.codento.com/2012/02/hello-ec2-part-1-bootstrapping-instances-with-cloud-init-git-and-puppet/