ECS Failed to create service due to assume role

Question

I got the following error when attempting to create an ECS service (Fargate) using Cloud Formation.

Invalid request provided: CreateService error: Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions. (Service: Ecs, Status Code: 400, Request ID: 32dc55bc-3b69-46dd-bf95-f3fff77c2508, Extended Request ID: null)

Things that tried/related:

Updating the role to include even AdministratorAccess (just for troubleshooting).
Allowing several services (ecs, elb, ec2, cloudformation) to assume role (was only ecs-tasks originally).
Create ECS service in web console successfully (same config). (But Cloud Formation doesn't work).
The ECS role has not been updated, the last successful ECS service creation was 21 Nov 2020 (/w Cloud Formation)

The following is the ECS role and Cloud Trail event of the above error. Has anyone faced similar issues or know what is happening?

Edit 1: ECS template is included, IAM role and the ECS service belongs to different root stack such that it is not possible to use DependsOn attribute. We have CI/CD that ensures the IAM stack is updated before the ECS stack.

ECS Task role used:

  EcsTaskRole:
    Type: 'AWS::IAM::Role'
    Properties:
      ManagedPolicyArns:
        - 'arn:aws:iam::aws:policy/AdministratorAccess'
        - 'arn:aws:iam::aws:policy/AmazonSQSFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonS3FullAccess'
        - 'arn:aws:iam::aws:policy/AmazonSNSFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonDynamoDBFullAccess'
        - 'arn:aws:iam::aws:policy/AmazonRDSFullAccess'
        - 'arn:aws:iam::aws:policy/service-role/AmazonECSTaskExecutionRolePolicy'
        - 'arn:aws:iam::aws:policy/AmazonSSMReadOnlyAccess'
        - 'arn:aws:iam::aws:policy/AWSXrayFullAccess'
        - 'arn:aws:iam::aws:policy/AWSBatchFullAccess'
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
          - Effect: Allow
            Principal:
              Service:
                - ecs-tasks.amazonaws.com
                - ecs.amazonaws.com
                - cloudformation.amazonaws.com
                - elasticloadbalancing.amazonaws.com
                - ec2.amazonaws.com
            Action:
              - 'sts:AssumeRole'
Outputs:
  EcsTaskRoleArn:
    Description: EcsTaskRoleArn
    Value: !GetAtt EcsTaskRole.Arn
    Export:
      Name: !Sub "${AWS::StackName}-EcsTaskRoleArn"

Event from Cloud Trail: (Masked some info)

{
   "eventVersion":"1.08",
   "userIdentity":{
      "type":"IAMUser",
      "principalId":"********",
      "arn":"arn:aws:iam::*****:user/****",
      "accountId":"*********",
      "accessKeyId":"********",
      "userName":"********",
      "sessionContext":{
         "sessionIssuer":{
            
         },
         "webIdFederationData":{
            
         },
         "attributes":{
            "mfaAuthenticated":"false",
            "creationDate":"2021-01-01T20:48:02Z"
         }
      },
      "invokedBy":"cloudformation.amazonaws.com"
   },
   "eventTime":"2021-01-01T20:48:14Z",
   "eventSource":"ecs.amazonaws.com",
   "eventName":"CreateService",
   "awsRegion":"ap-east-1",
   "sourceIPAddress":"cloudformation.amazonaws.com",
   "userAgent":"cloudformation.amazonaws.com",
   "errorCode":"InvalidParameterException",
   "errorMessage":"Unable to assume role and validate the specified targetGroupArn. Please verify that the ECS service role being passed has the proper permissions.",
   "requestParameters":{
      "clientToken":"75e4c412-a82c-b01a-1909-cfdbe788f1f1",
      "cluster":"********",
      "desiredCount":1,
      "enableECSManagedTags":true,
      "enableExecuteCommand":false,
      "healthCheckGracePeriodSeconds":300,
      "launchType":"FARGATE",
      "loadBalancers":[
         {
            "targetGroupArn":"arn:aws:elasticloadbalancing:ap-east-1:********:listener-rule/app/********/e6a62b4cc4d13aaa/098a6759b6062f3f/f374eba8a4fb66e5",
            "containerName":"********",
            "containerPort":8080
         }
      ],
      "networkConfiguration":{
         "awsvpcConfiguration":{
            "assignPublicIp":"ENABLED",
            "securityGroups":[
               "sg-025cd908f664b25fe"
            ],
            "subnets":[
               "subnet-067502309b0359486",
               "subnet-018893d9e397ecac5",
               "subnet-0bfb736aefb90f05a"
            ]
         }
      },
      "propagateTags":"SERVICE",
      "serviceName":"********",
      "taskDefinition":"arn:aws:ecs:ap-east-1:********:task-definition/********"
   },
   "responseElements":null,
   "requestID":"32dc55bc-3b69-46dd-bf95-f3fff77c2508",
   "eventID":"3f872d94-72a7-4ced-96a6-028a6ceeacba",
   "readOnly":false,
   "eventType":"AwsApiCall",
   "managementEvent":true,
   "eventCategory":"Management",
   "recipientAccountId":"904822583864"
}

Cloud formation template of ECS service


  MyServiceLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: my-service-log
      RetentionInDays: 365
  MyServiceTargetGroup:
    Type: 'AWS::ElasticLoadBalancingV2::TargetGroup'
    Properties:
      HealthCheckPath: /my-service/health
      HealthCheckIntervalSeconds: 300
      HealthCheckTimeoutSeconds: 10
      Name: my-service-target-group
      TargetType: ip
      Port: 8080
      Protocol: HTTP
      VpcId: !Ref VpcId
  MyServiceListenerRule:
    Type: 'AWS::ElasticLoadBalancingV2::ListenerRule'
    Properties:
      Actions:
        - Type: forward
          TargetGroupArn: !Ref MyServiceTargetGroup
      Conditions:
        - Field: path-pattern
          Values:
            - /my-service/*
      ListenerArn: !Ref AppAlbListenerArn
      Priority: 164
  MyServiceTaskDef:
    Type: 'AWS::ECS::TaskDefinition'
    Properties:
      ContainerDefinitions:
        - Name: my-service-container
          Image: !Join
            - ''
            - - !Ref 'AWS::AccountId'
              - .dkr.ecr.
              - !Ref 'AWS::Region'
              - .amazonaws.com/
              - 'Fn::ImportValue': !Sub '${RepositoryStackName}-MyServiceECR'
              - ':'
              - !Ref MyServiceVersion
          Essential: true
          PortMappings:
            - ContainerPort: 8080
              Protocol: tcp
          LogConfiguration:
            LogDriver: awslogs
            Options:
              awslogs-group: !Ref MyServiceLogGroup
              awslogs-region: !Ref AWS::Region
              awslogs-stream-prefix: my-service
      RequiresCompatibilities:
        - FARGATE
      Cpu: 256
      Memory: 512
      Family: my-service-taskdef
      NetworkMode: awsvpc
      ExecutionRoleArn:
        'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
      TaskRoleArn:
        'Fn::ImportValue': !Sub '${IamStackName}-EcsTaskRoleArn'
      Volumes: []
  MyServiceECS:
    Type: 'AWS::ECS::Service'
    Properties:
      DesiredCount: 1
      Cluster: !Ref EcsCluster
      TaskDefinition: !Ref MyServiceTaskDef
      LaunchType: FARGATE
      NetworkConfiguration:
        AwsvpcConfiguration:
          AssignPublicIp: ENABLED
          SecurityGroups:
            - !Ref SecurityGroupECS
          Subnets:
            - !Ref DmzSubnet1
            - !Ref DmzSubnet2
            - !Ref DmzSubnet3
      LoadBalancers:
        - ContainerName: my-service-container
          ContainerPort: '8080'
          TargetGroupArn: !Ref MyServiceListenerRule
      EnableECSManagedTags: true
      PropagateTags: SERVICE
      HealthCheckGracePeriodSeconds: 300
    DependsOn:
      - MyServiceListenerRule

How and where do you use EcsTaskRole role? Can you show the the rest of your template? — Marcin
Another root stack, the IAM role and ECS service are not under the same root stack — Nick97
@Marcin Thanks for helping. Just updated on the question itself. — Nick97

Asri Badlah Asri Badlah · Accepted Answer · 2021-01-01T21:38:10

Use the DependsOn attribute to specify the dependency of the AWS::ECS::Service resource on AWS::IAM::Policy.

ECS Failed to create service due to assume role

2 Answers