CloudFormation wait condition in a update stack

11
votes

I have a cloud formation template that I set up a wait condition to recieve a signal when the user data script finish to execute. This works perfect!

Now I'm using cfn-hup to update my application with a new version. I update the stack with a new parameter and a script download the new version and install it on the server. But as I don't update the wait condition resource, it is not recreated and it do not wait until the signal. Is there anyway to force the wait condition resource recreation?

3
amazon-web-servicesamazon-cloudformation
I don't think that's possible. What do you want to do if the new wait condition fails? Do you want to delete the stack? – Edwin
No, I want it to be rollbacked. It works if I create a new wait condition with a different logic name but I don't want to do that because I would need to change the template. I just would like to force the wait condition to be recreated every update or if a bit ec2 instance is being updated. – Luiz Guilherme
Fantastic question. With the support of "UpdatePolicy" on the AutoScaleGroup resources, which can create new resources the same problem exists. The creation of the stack follows the appropriate ordering however we are unable to dynamically generate a new WaitCondition to ensure that not only is the new box "running" but has been properly provisioned via the UserData block before starting the rolling update of the other resources further down the dependency chain. If I get it working with ASG I will comment again on this, as it is a very similar approach. – jnt30

3 Answers

3
votes

Great question. I have been searching for a solution myself, however it does appear that this is not possible. From a 2013 re:Invent session:

  • cfn-hup cannot interact with CloudFormation workflow
    • Workflow will not wait for cfn-hup
    • cfn-hup cannot fail workflow
    • cfn-hup cannot inject data into stack

Source: http://www.slideshare.net/AmazonWebServices/aws-cloudformation-under-the-hood-dmg303-aws-reinvent-2013-28437139/72

"If cfn-hup crashes and burns, and fails miserably, the CloudFormation workflow will say update complete."

Source: https://www.youtube.com/watch?v=ZhGMaw67Yu0&t=36m39s

1
votes

To update an application with CloudFormation, you update data in the "Metadata" section of an Instance or LaucnConfig item. In your case, you change the value of a parameter which is referenced in the "Metadata" section. I'm guessing that a "source" is being updated? A cron job on your instance(s) is set to run cfn-hup, and when it does, it reads the information in the CloudFormation stack and sees that it has changed. Then it performs the actions specified in the hooks.conf file (or those specified in files in the hooks.d directory), which is typically to run cfn-init. The command cfn-init retrieves all the files from the sources specified, and runs all the commands that are specified in the "Metadata" section. The result is an updated app.

For the wait condition to "reactivate" something would have to tell it to, which, btw, I don't think this is possible. Since the only thing you are changing is Instance metadata, and all actions are happening local to the instance, I don't think what you are trying to do would be possible. A new wait condition would have to be created and signaled by cfn-signal (which would have to be called in hooks.conf, I believe). An interesting scenario: What if the following occurred: You rename the wait condition and change the metadata for the update. The instance fails to signal complete and CloudFormation rolls back the template to the previous state. On the next call to cfn-hup, the instance sees that the stack has changed, but the reversion hangs and again fails to signal complete. Would we be stuck in an infinite loop?

1
votes

thanks for sharing, I encountered the similar problem, and I've created my own workaround, even it's not nice but I'm happy to share it here:

The key problem here is cfn-hup runs asynchronously, and there's no way for us to hook it back to cloudformation.

Here's my solution: - as part of cfn-auto-reload.conf, - we invoke the cfn-init cmd to update stack, - and we use another script to capture the exit code of cfn-init cmd and the version of installed app, - then put them in a file on S3.

  • then as a follow up work of deployment on CI pipeline, a verification task will check this file on S3, to verify both the exit code and version.

not elegant but it works :)