Why Celery queues randomly stuck on starting using Supervisord?

Question

There are over 20 workers managed by supervisord. My celery worker command:

celery worker myproject.server.celery -l INFO --pool=gevent --concurrency=10 --config=myproject.celeryconfig -n default_worker.%%h -Q default

the problem is: each time deploy new code and then restart each supervisor task, few workers would stuck on staring randomly, which is confused. You can check the Flower dashboard and found the stuck worker: Image: flower dashboard worker status

Then, you can find the more strange in htop, the ldconfig.real started, instead of the failed celery worker: Image: htop monitor celery worker

I appreciate any suggestion!

Answer 1

If you are gracefully restarting, then this is a normal behaviour because at the time of deployment you may have some active tasks. Celery will not kill the worker processes when in the graceful shutdown state. It will wait for all of them to finish, and then stop.