EC2 Container Service vs Apache Mesos

2
votes

We are looking to use Docker container to run our batch jobs in a cluster enviroment. We are evaluating to use AWS ECS Container Service/Chronos/Mesos. As far as I know, Apache Mesos has some overlapping features/purpose that EC2 has, like cluster management. Chronos is a distributed scheduler.

I am having dificult to correlate all this technologies to create a architecture!

EC2 service replace Mesos? What about the scheduler?

We are a small team will little experience in cluster development. Which stack is better for our batch processing?

EDIT I make a huge edit, and i think now i understand the architecture:

enter image description here

This is a sample picture with two cluster been managed by Mesos.

Reading the ECS Container Service documentation(http://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling_tasks.html), AWS is on the way of integrete ECS with the Mesos Apache Framework. So I imagine that using in the future, we can use the mesos framework to manage the resources in the ECS Cluster. So it is going to be possible to use Chronos (for batch scheduling) and Marathon (for long running app.)

EDIT At this moment, we dont have distributed jobs running, like hadoop jobs or sparks jobs. Our job are much simpler, running on single instances of EC2. We are planning to use Docker to run our batch running jobs.

1
amazon-ec2cronschedulermesos
Pedro, thanks for putting the time in here to understand the problem space, however, it sucks to answer a question that is an ever-moving target. You've updated your question a couple of times now, substantially changing it and making my answer below looking totally off-topic. I suggest you either create a new question or somehow indicate what you think about my answer (or point out what is missing, in a comment). – Michael Hausenblas
BTW, the architecture is in so far wrong that Hadoop runs on Mesos (not the other way round), see github.com/mesos/hadoop and github.com/mesosphere/hdfs – Michael Hausenblas
For sure! I will open another post. The picture only says that Mesos can control worker node of heterogeneous cluster types. – p.magalhaes
Ok, fine. Would have been cool to ack my answer below as well but guess I'll wait until you post the new one :/ – Michael Hausenblas
I am going to study a little more to post again. :) – p.magalhaes

1 Answers

5
votes

I'd argue it depends on the type of batch jobs, but the Apache Mesos ecosystem is certainly more flexible than ECS to accomodate your needs. The flexibility comes from the fact that Mesos uses a so called two-level scheduling model, which is a fancy name for it outsources the scheduling decision into frameworks (rather than trying to implement each and every existing and future workload scheduling strategy in its core, itself).

You mentioned one such a framework already, Chronos, which is a good working horse, just maybe don't use the dependencies for jobs, ok? Then there is another great batch framework called Cook. Depending on your needs (for example SQL-based batch report generation) you could use Apache Spark. And so on and so forth.

BTW, did I mention already that with Mesos you don't risk a vendor lock-in, while being able to deploy it, depending on your needs, fully in one cloud (such as AWS), hybrid cloud (say AWS and GCP/Azure) or on-premises?

UPDATE: to clarify, of course Mesos has first-class Docker support.