How to update a CloudFormation stack without updating an ECS service task definition

Question

I have a CloudFormation stack which stands up an entire environment for our application (including VPCs, subnets, security groups, roles, lambda functions, load balancers, S3 buckets and CloudFront distributions).

In addition to these things, it also creates an ECS cluster with an ECS service with an initial task definition:

Resources:
  # ...snip...
  Cluster:
    Type: AWS::ECS::Cluster
    Properties:
      ClusterName: !Sub 'cluster-${Environment}'
  APITaskDefinition:
    Type: AWS::ECS::TaskDefinition
    DependsOn:
      - APIExecutionRole
    Properties:
      Family: !Sub 'api-${Environment}'
      Cpu: 512 
      Memory: 1024
      ExecutionRoleArn: !Ref APIExecutionRole
      NetworkMode: awsvpc
      RequiresCompatibilities:
        - FARGATE
      ContainerDefinitions:
        # ...snip...
  APIService:
    Type: AWS::ECS::Service
    DependsOn:
      - LoadBalancerListenerApiHttps
      - TargetGroupApi
    Properties:
      ServiceName: !Sub 'service-${Environment}-api'
      Cluster: !Ref Cluster
      TaskDefinition: !Ref APITaskDefinition
      LaunchType: FARGATE
      DeploymentConfiguration:
        MinimumHealthyPercent: 100
        MaximumPercent: 200
      DesiredCount: 1
      EnableECSManagedTags: true
      PropagateTags: SERVICE
      # ...snip...

The above template sets the task definition for the service to an initial/placeholder task, but once the environment has been created, we deploy new versions of our application to ECS (using the AWS CLI) which involves creating new task definitions and updating the ECS service to use the new task definition.

However, when I go to make a non-ECS related change to the CloudFormation stack (e.g. changing one of the properties of a CloudFront distribution) and create a change-set, it always resets the ECS service to use the initial task definition defined in the template.

Is there anyway I can tell CloudFormation not to update the ECS service when performing stack updates? I've tried using a stack policy to prevent updates to the service but this just causes the whole update operation to fail.

Can you perform drift detection before update? Maybe !Ref APITaskDefinition is pointing to different task definition version that it is currently running by the service? — Marcin
@Marcin Yes it is. The CloudFormation template task definition points to an initial/placeholder task definition for setting up a new environment, but the service itself gets a new task definition every time we update our API. — DanielGibbs
Thus you have to update the CFN tempate to reflect the change, or fully decouple ECS service from the main template. You can keep it in a separate template so that it does not get updated when the CF is updated. — Marcin
@Marcin Ok, so essentially I can't do what I'm trying to do with an existing stack? — DanielGibbs
I'm not aware of a technique to do this in an another way. Generally you shoudn't be updating your ECS outside of CFN. Because you do update the service independently, you have a drift. Let me know how it goes. If it will work out, I could provide an answer with summary. — Marcin

scampbell scampbell · Accepted Answer · 2020-09-21T11:44:43

Store the latest deployed image reference in the SSM parameter store and update this SSM param after a successful deployment using your CLI deployment process.

Then reference the SSM parameter as a parameter in CloudFormation e.g.

Parameters:
  ContainerDefaultImage:
    Type: AWS::SSM::Parameter::Value<String>
    Description: The name of the parameter containing the image definition file to be used.
    Default: /some/param/name

CloudFormation will resolve the SSM parameter value on template deployment.

I use this technique to allow different container versions across different environment while still using the same template. My containers are built and deployed in a different pipeline and different environments have different versions deployed for testing, UAT etc.

How to update a CloudFormation stack without updating an ECS service task definition

1 Answers