Why can't I run multiple tasks in AWS ECS?

Question

I'm working on a file processing system where files can be uploaded to S3 and then processed in a container. I have been using triggering ECS to run tasks from lambda and passing a few environment variables.

S3 -> Lambda -> ECS

I'm running into a problem where I can't seem to run more than 1 task at once. If a task is already running then any subsequent tasks that get run are stuck in "PROVISIONING" and eventually disappear altogether.

Here is my lambda function that runs the ECS task:

const params: RunTaskRequest = {
    launchType: "FARGATE",
    cluster: "arn:aws:ecs:us-east-1:XXXXXXX:cluster/FileProcessingCluster",
    taskDefinition: "XXX",
    networkConfiguration: {
        awsvpcConfiguration: {
            subnets: [
                "subnet-XXX",
                "subnet-XXX"
            ],
            securityGroups: [
                "..."
            ],
            assignPublicIp: "DISABLED"
        }
    },
    overrides: {
        containerOverrides: [
            {
                name: "FileProcessingContainer",
                environment: [
                    ...
                ]
            },
        ]
    },
};

try {
    await ecs.runTask(params).promise(); 
}catch (e) {
    console.error(e, e.stack)       
}

I'm using AWS-CDK to create the ECS infrastructure:

    const cluster = new ecs.Cluster(this, 'FileProcessingCluster', {
        clusterName: "FileProcessingCluster"
    });

    const taskDefinition = new ecs.FargateTaskDefinition(this, "FileProcessingTask", {
        memoryLimitMiB: 8192,
        cpu: 4096,
    });

    taskDefinition.addContainer("FileProcessingContainer", {
        image: ecs.ContainerImage.fromAsset("../local-image"),
        logging: new ecs.AwsLogDriver({
            streamPrefix: `${id}`
        }),
        memoryLimitMiB: 8192,
        cpu: 4096,
    });

Is there some something I'm missing here? Perhaps a setting related to concurrent tasks?

Can you check if there are any details on the problem in the ECS event log? docs.aws.amazon.com/AmazonECS/latest/developerguide/… — Mark B
Just a lot of 'service my-service has reached a steady state.' events — Harry
I believe could be due to a misconfiguration of scaling options — Harry
Are you starting an ECS "service" or a "task"? Given your description, it sounds like you should be using tasks, not a service, and not setting any scaling settings at all. If you were sending S3 events to SQS, the you could use a service that auto-scales based on the number of items in the queue. You should probably add your Lambda code to your question. — Mark B
I'm starting tasks, but I also have a service. If I'm honest I don't completely understand the whole thing given I just started with it yesterday. So you're saying I don't need a service and could simply run tasks? — Harry

Harry Harry · Accepted Answer · 2020-11-16T19:12:51

It turns out that I misconfigured the subnets in the task definition, this was preventing the image pull from ECR.

You can read more about it here: ECS task not starting - STOPPED (CannotPullContainerError: “Error response from daemon request canceled while waiting for connection”

And: https://aws.amazon.com/premiumsupport/knowledge-center/ecs-pull-container-error/

Why can't I run multiple tasks in AWS ECS?

1 Answers