Set the ECS Cloudformation Update Stack timeout?

6
votes

When updating a Cloudformation EC2 Container Service (ECS) Stack with a new Container Image, is there any way to control the timeout so if the service does not stabilize it rolls back automatically?

The UpdatePolicy attribute which is part of the Auto Scaling Group does not help since instances are not being created.

I also tried a WaitCondition but have not been able to get that to work.

The stack essentially just stays in the UPDATE_IN_PROGRESS state until it hits the default timeout (~3 hours), or you trigger a Cancel the update.

Ideally we would be able to have the stack timeout after a short period of time.

This is what my Cloudformation template looks like: https://s3.amazonaws.com/aws-rga-cw-public/ops/cfn/ecs-cluster-asg-elb-cfn.yaml
Thanks.

4
amazon-cloudformationamazon-ecs
Ideally, I wish that both create-stack and update-stack had a --timeout-in-minutes option. Unfortunately only create-stack has this. – Nate
@Nate yes, exactly. that would be indispensable – nauman hafiz
We run into the same issue and have yet to find a proper solution :( AWS directed us to use the WaitCondition, but it is kinda tricky because what will happen if the timeout is set to 45 minutes but the update takes more then 45 minutes? – tvb

4 Answers

3
votes

I've created a workaround for this problem until AWS creates a ECS UpdatePolicy and CreationPolicy that allows for resourcing signaling:

Use AWS::CloudFormation::WaitCondition with a Macro that will create new WaitCondition resources when the service is expected to update. Signal the wait condition with a non-essential container attached to the task.

Example: https://github.com/deuscapturus/cloudformation-macro-WaitConditionUpdate/blob/master/example-ecs-service.yaml

The Macro for the above example can be found here: https://github.com/deuscapturus/cloudformation-macro-WaitConditionUpdate

1
votes

If your WaitCondition is in the original create you need to rename it (and the Handle). Once a waitcondition has been signaled as complete, it will always be complete. If you rename it and do an update, the original WaitCondition and Handle will be dropped and the new ones created created and signaled.

If you don't want to have to modify your template you might be able to use Lamba and Custom resources to create a unique WaitCondition via the aws cli for each update.

1
votes

My workaround for this problem is that before triggering an update stack, run a script in the background

./deployment-breaker.sh &

And for the script

#!/bin/bash
sleep 600
$deploymentStatus = (aws cloudformation describe-stack --stack-name STACK_NAME | jq XXX)
if [[ $deploymentStatus == YOUR_TERMINATE_CONDITION ]]then
  aws cloudformation cancel-update-stack --stack-name STACK_NAME
fi
0
votes

It's not possible at the moment with the provided CloudFormation types. I have the same problem and I might create a custom CloudFormation resource (usineg AWS Lambda) to replace my AWS::ECS::Service.

The other alternative is to use nested stacks to wrap the AWS::ECS::Service resources — it won't solve the problem, but it at least will isolate the individual service and the rest of the stack will be in a good state. My stacks have multiple services and this would help, but the custom resource is the best option so far (I know other people that did the same thing).