After updating Fargate TaskDefinition, CloudWatch events that trigger tasks fail because of inactive task definitions

Question

I have a series of tasks defined in ECS that run on a recurring schedule. I recently made a minor change to update my task definition in Terraform to change default environment variables for my container (from DEBUG to PRODUCTION):

        "environment": [
              {"name": "ENVIRONMENT", "value": "PRODUCTION"}
              ]

I had this task running using the Scheduled Tasks feature of Fargate, setting it at a rate of every 4 hours. However, after updating my task definition, I began to see that the tasks were not being triggered by CloudWatch, since my last container log was from several days ago.

I dug deeper into the issue using CloudTrail, and noticed one particular part of the entry for a RunTask event:

"eventTime": "2018-12-10T17:26:46Z",
"eventSource": "ecs.amazonaws.com",
"eventName": "RunTask",
"awsRegion": "us-east-1",
"sourceIPAddress": "events.amazonaws.com",
"userAgent": "events.amazonaws.com",
"errorCode": "InvalidParameterException",
"errorMessage": "TaskDefinition is inactive",

Further down in the log, I noticed that the task definition ECS was attempting to run was

 "taskDefinition": "arn:aws:ecs:us-east-1:XXXXX:task-
definition/important-task-name:2",

However, in my ECS task definitions, the latest version of important-task-name was 3. So it looks like the events are not triggering because I am using an "inactive" version of my task definition.

Is there any way for me to schedule tasks in AWS Fargate without having to manually go through the console and stop/restart/update each cluster's scheduled update? Isn't there any way to simply ask CloudWatch to pull the latest active task definition?

Matheus Ianzer Matheus Ianzer · Accepted Answer · 2020-04-13T19:00:22

You can use CloudWatch Event Rules to control scheduled tasks and whenever you update a task definition you can also update your rule. Say you have two files:

myRule.json

{
  "Name": "run-every-minute",
  "ScheduleExpression": "cron(0/1 * * * ? *)",
  "State": "ENABLED",
  "Description": "a task that will run every minute",
  "RoleArn": "arn:aws:iam::${IAM_NUMBER}:role/ecsEventsRole",
  "EventBusName": "default"
}

myTargets.json

{
  "Rule": "run-every-minute",
  "Targets": [
      {
          "Id": "scheduled-task-example",
          "Arn": "arn:aws:ecs:${REGION}:${IAM_NUMBER}:cluster/mycluster",
          "RoleArn": "arn:aws:iam::${IAM_NUMBER}:role/ecsEventsRole",
          "Input": "{\"containerOverrides\":[{\"name\":\"myTask\",\"environment\":[{\"name\":\"ENVIRONMENT\",\"value\":\"production\"},{\"name\":\"foo\",\"value\":\"bar\"}]}]}",
          "EcsParameters": {
            "TaskDefinitionArn": "arn:aws:ecs:${REGION}:${IAM_NUMBER}:task-definition/myTaskDefinition",
            "TaskCount": 1,
            "LaunchType": "FARGATE",
            "NetworkConfiguration": {
                "awsvpcConfiguration": {
                    "Subnets": [
                        "subnet-xyz1",
                        "subnet-xyz2",
                    ],
                    "SecurityGroups": [
                        "sg-xyz"
                    ],
                    "AssignPublicIp": "ENABLED"
                }
            },
            "PlatformVersion": "LATEST"
          }
      }
  ]
}

Now, whenever there's a new revision of myTaskDefinition you may update your rule, e.g.:

aws events put-rule --cli-input-json file://myRule.json --region $REGION
aws events put-targets --cli-input-json file://myTargets.json --region $REGION
echo 'done'

But of course, replace IAM_NUMBER and REGION with your credentials,

After updating Fargate TaskDefinition, CloudWatch events that trigger tasks fail because of inactive task definitions

2 Answers