AWS ECS running a task that requires many cores

2
votes

I am conceptually trying to understand how to use AWS ECS to run my "cluster" jobs.

I have some scientific software inside a Docker container, that natively takes advantage of as many cores as the underlying instance has to offer.

My question in this case is, can I use AWS ECS to "increase" the number of "visible" cores to the task running inside my Docker container. For instance, is my "cluster" limited to only a single instance? Or is a "cluster" expandable to multiple instances?

I haven't been able to find any answers my looking through he AWS docs.

3
amazon-web-servicesdockeramazon-ecs

3 Answers

3
votes

Cluster is just some EC2 instances that are ECS-enabled (are running special agent software) and grouped together. Tasks that you run on this cluster are spread across these instances. Each task can involve multiple containers. However, each container stays within its instance ‘boundaries’, hardware-wise. It is allocated a number of “CPU units” and shares them with other containers running on the same instance.

From my understanding, running a process spanning multiple cores in a container is not quite fitting ECS architecture idea—it seems like trying to do part of ECS’s scheduler job.

I found these resources useful when I was reading about it:

2
votes

I had a similar situation moving a Python app that used a script to spawn copies of itself based on the number of cores. The answer to this isn't so much an ECS problem as it is a Docker best practice... you should strive to use 1 process per container. (see https://docs.docker.com/engine/userguide/eng-image/dockerfile_best-practices/)

How I ended up implementing this was using a Dockerfile to run each process and then used essential ECS tasks so it will reload itself if the task died.

Your cluster is a collection of EC2 instances with the ECS service running. Each instance has a certain number of CPU 'units' (typically 1024 units === 1 core) and RAM. I profiled my app at peak load and tweaked the mix until I got it where I liked it. If your app can use more CPU than that, try giving it 2048 CPU or some other amount and see how it performs. I used Meros (https://meros.io/) to profile my app.

Hope this helps!

0
votes

"increase" the number of "visible" cores to the task running inside my Docker container

Container and cluster is different things, you may run lot of containers on one instance, but you can't run one container on multiply instances.

Cluster - it is set of docker containers.

is my "cluster" limited to only a single instance?

no, you may choose number of instances in cluster