Django + Celery on Amazon AWS - Using separate EC2 instances as workers

Question

I have a Django application. I am using Celery to run long-running processes on the background. Both the application and celery workers run on the same machine.

Now we are moving our servers to AWS. On AWS, we want to create a setup like the following:

We have n EC2 instances that run the app servers, and we have m EC2 instances as workers. When we need to do a long-running process, app server sends this job to the worker, and the worker processes the job. But the job is dependent on Django models and the database.

How can we setup the workers to enable them run these django model dependent jobs?

Answer 1

This is not AWS specific.

You have to:

1. make sure every server has same version of app code
2. all workers spread across servers use same task broker and result backend
3. workers can connect to your DB (if it's needed)

More verbose advise for config needs additional info :)

Answer 2

Another approach to this would be to use EC2 Container Service, with two different running docker containers, one for the app and one for the worker.