Updating Cloudformation Stack with ECS Task Definition and Service results in hung stack

2
votes

Im having a repeated issue when i run Update Stack on a Cloudformation stack consisting of ECS resources such as Task Definition and Service, the Stack hangs and cannot be killed until it times out altogether.

I am simply updating the Task with a new Tag of the Container Image, but it just hangs after running for a few minutes, and the updated Task never gets added to the Service.

Any ideas? Is this a bug in ECS?

EDIT - @wjordan here is the cloudformation template: https://s3.amazonaws.com/rga-g1-public/ops/cfn/templates/ecs-cluster-asg-elb-cfn.yaml

@GigaPalmer I added DeployConfiguration to the Service Resource with maxpercentage as 200 and min set to 50, but that did not fix the issue either.

The initial build of the stack always works correctly, but as soon as i trigger an update it seems to fail 3 out of 4 times, all of which end up with a hung stack in UPDATE mode.

1
functionamazon-web-servicestaskamazon-ecsamazon-cloudformation
This may be related to the attribute: DeploymentConfiguration which tells you how many task to run for the deployment. docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/… – Palmer
Please provide a Minimal, Complete and Verifiable example, such as the full contents of your CloudFormation stack template, so that others can reproduce/verify your issue. – wjordan
sure, i am still in the process of verifying if adding DeploymentConfiguration fixes the issue. But the template is available for here: cfn template @wjordan – nauman hafiz
How did you update your task definition, when i update my stack it deregisters the current task definition and makes it inactive. There is only one task definition which is active at any time when i use CFT. – sks123245

1 Answers

0
votes

Looks like the issue here was in the app itself. We had a python process running under PM2 that would end up as a stuck process and ECS was unable to determine the health of the service.

This issue can be closed.