AWS ECS EC2: TaskCanceledException when calling AWS API (connection timed out)

Question

I have set up an AWS ECS cluster with EC2-type container instances. In task definition, there is "SECRETS" environment variable specified with the value corresponding to a particular secret name. Task definition uses awsvpc network mode.

In order to access secrets value from code (.net) following code (from aws snippet) is used:

IAmazonSecretsManager client = new AmazonSecretsManagerClient(region);
GetSecretValueRequest request = new GetSecretValueRequest
{
  SecretId = secretName,
  VersionStage = "AWSCURRENT" // VersionStage defaults to AWSCURRENT if unspecified.
};
GetSecretValueResponse response = Task.Run(async () => await client.GetSecretValueAsync(request)).Result;

This works perfectly with Fargate instance type. When switching to EC2 container instance GetSecretValueAsync() fails with AggregateException : TaskCanceledException.

I've tried to get IAM role credentials from the inside of the container with success:

curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI

I've also tried specifying retrieved credentials directly with no luck:

AmazonSecretsManagerClient(awsAccessKeyId, awsSecretAccessKey, region)

In addition, I've tried baking aws cli inside the container, and from the inside I've tried aws secretsmanager , aws iam get-user and aws sts get-caller-identity - still hangs without response. I've granted full admin access to task execution role - still no success. I'm able to retrieve secrets from EC2 container instance, but not from the mounted container.

A long shot, but are you using the latest version of the ECS AMI? — mcheshier
@mcheshier I've tried new LC with updated AMI with no luck. In addition, I've tried baking aws cli inside the container, and from the inside aws secretsmanager, aws iam get-user or even aws sts get-caller-identity hangs without response. Looks like there is some IAM permissions, I've missed. — Sergey Nikitin

Sergey Nikitin Sergey Nikitin · Accepted Answer · 2019-06-27T17:40:08

Thanks to AWS support, the solution was found. The key issue of my configuration was a combination of awsvpc networking mode and EC2 launch type for containers:

The awsvpc network mode does not provide task ENIs with public IP addresses for tasks that use the EC2 launch type. To access the internet, tasks that use the EC2 launch type must be launched in a private subnet that is configured to use a NAT gateway

Instead, I've moved to bridge networking with dynamic port mapping (achieved by using Application Load Balancer). I've also used host networking mode for some specific tasks - this worked as well.

AWS ECS EC2: TaskCanceledException when calling AWS API (connection timed out)

1 Answers