Celery: how to assign some specific number of workers to different queues?

0
votes

I have a celery app consuming from large amount of queues with gevent concurrency. For each queue I need to have some guaranteed number of workers, but the number is different for each queue. For example:

  • qA must have 10
  • qB must have 25
  • qN must have M etc.

I'd like to avoid to start dedicated celeryd for each queue due to the large total amount of queues.

Workers do a lot of IO and I use -P gevent so don't care how many subworkers celeryd starts.

Is there a way to configure a single celeryd instance to assign some specific number of workers to different queues?

1
pythoncelerygeventcelery-taskceleryd

1 Answers

0
votes

Celery multi allows to configure workers independently of each other. For example

celery multi start node1 node2 -c:1 5 -Q:1 queue1 -c:2 10 -Q:2 queue2

starts 2 workers: the first node1 consumes from queue1 having concurrency 5, and the second node2 consumes from queue2 having concurrency 10.

Have a look at multi.py docstring