How do celery workers communicate in Heroku

0
votes

I have some celery workers in a Heroku app. My app is using python3.6and django, these are the relevant dependencies and their versions:

  • celery==3.1.26.post2
  • redis==2.10.3
  • django-celery==3.2.2

I do not know if the are useful to this question, but just in case. On Heroku we are running the Heroku-18 stack.

As it's usual, we have our workers declared in a Procfile, with the following content:

web: ... our django app ....
celeryd: python manage.py celery worker -Q celery --loglevel=INFO -O fair
one_type_of_worker: python manage.py celery worker -Q ... --maxtasksperchild=3 --loglevel=INFO -O fair
another_type: python manage.py celery worker -Q ... --maxtasksperchild=3 --loglevel=INFO -O fair

So, my current understanding of this process is the following:

Our celery queues run on multiple workers, each worker runs as a dyno on Heroku (not a server, but a “worker process” kind of thing, since servers aren’t a concept on Heroku). We also have multiple dynos running the same celery worker with the same queue, which results in multiple parallel “threads” for that queue to run more tasks simultaneously (scalability). The web workers, celery workers, and celery queues can talk to each other because celery manages the orchestration between them. I think it's specifically the broker that handles this responsibility. But for example, this lets our web workers schedule a celery task on a specific queue and it is routed to the correct queue/worker, or a task running in one queue/worker can schedule a task on a different queue/worker.

Now here is when comes my question, so does the worker communicate? Do they use an API endpoint in localhost with a port? RCP? Do they use the broker url? Magic?

I'm asking this because I'm trying to replicate this setup in ECS and I need to know how to set it up for celery.

pythondjangoherokucelerydjango-celery