Celery workers wait

2
votes

I am writing an application with using Celery framework. Some of my tasks are pretty heavyweight and can execute for a long time.

I've noticed that when I run 5-6 workers and then put 10-20 tasks they may be distributed by workers randomly and sometimes if one get free of tasks, it does not start remaining ones and they will be handled by others only when they complete their tasks (maybe in hours). If I run one more worker at this time - it does nothing, but can accept new tasks.

Is it a bug or a feature and how do I solve my needs? It does not make sense to wait hours while we have free workers and not started tasks.

1
pythoncelery

1 Answers

1
votes

It is not a bug or a feature (more likely a feature), it is just misconfiguration.

As the documentation says, the worker can reserve some tasks for himself to hasten the processing messages. But this makes sense only for small and fast tasks - it does not ask the broker for the new message but immediately starts reserved one.

But for the long tasks this may lead to the case described in your question.

If you have many tasks with a long duration you want the multiplier value to be 1, which means it will only reserve one task per worker process at a time.

If you have a combination of long- and short-running tasks, the best option is to use two worker nodes that are configured separately, and route the tasks according to the run-time.

So, you need to set CELERYD_PREFETCH_MULTIPLIER = 1 in the celery's settings.

But,

When using early acknowledgement (default), a prefetch multiplier of 1 means the worker will reserve at most one extra task for every active worker process.

When users ask if it’s possible to disable “prefetching of tasks”, often what they really want is to have a worker only reserve as many tasks as there are child processes.

I also may recommend to set CELERY_ACKS_LATE = True to send ACK command only after the task get completed. This way the worker won't reserve any additional tasks at all, but currently executing task will be marked as reserved only.

Although this has a side effect - if the worker get crashed/terminated in the middle of executing of your task, the task will be marked again as not-started and any other worker may start it again from the beginning. So make sure you have idempotent tasks. See docs again about this.