how to configure celery executing tasks concurrently from on queue

Question

In an environment with 8 cores, celery should be able to process 8 incoming tasks in parallel by default. But sometimes when new tasks are received celery place them behind a long running process.

I played around with default configuration, letting one worker consume from one queue.

celery -A proj worker --loglevel=INFO --concurrency=8

Is my understanding wrong, that one worker with a concurrency of 8 is able to process 8 tasks from one queue in parallel?

How is the preferred way to setup celery to prevent such behaviour described above?

Answer 1

To put it simply concurrency is the number of jobs running on a worker. Prefetch is the number of job sitting in a queue on a worker itself. You have 1 of 2 options here. The first is to set the prefetch multiplier down to 1. This will mean the worker will only keep, in your case, 8 additional jobs in it's queue. The second which I would recommend would be to create 2 different queues one for your short running tasks and another for your long running tasks.