Celery inspect shows worker offline when Concurrency value reached

Question

Our project uses Django to run a webserver with Celery/RabbitMQ doling out tasks to workers. Whenever a worker is at capacity for tasks (e.g. with concurrency of four has four tasks), they stop showing up from flower or the built in celery inspect.

ETA: The worker is still online, and when the tasks are completed such that the worker is below concurrency value, it appears again in flower and inspect. Unfortunately, I'm trying to use the 'reserved' method of inspect to see queued tasks, and this is always empty whenever I can see the worker.

Here are our Celery settings inside Django settings file:

BROKER_URL = 'amqp://[user]:[pw]@[host]//'

CELERY_ACCEPT_CONTENT = ['json']
CELERY_EVENT_SERIALIZER = 'json'
CELERY_TASK_SERIALIZER = 'json'
CELERY_RESULT_SERIALIZER = 'json'
CELERY_RESULT_BACKEND = 'amqp'
CELERY_TIMEZONE = 'US/Mountain'
CELERY_RESULT_PERSISTENT = True
CELERY_TASK_RESULT_EXPIRES = 3600 # seconds
CELERYBEAT_SCHEDULER = 'djcelery.schedulers.DatabaseScheduler'
CELERY_DEFAULT_QUEUE = 'tasks'
CELERY_TRACK_STARTED = True

from kombu import Queue

CELERY_QUEUES = (Queue('tasks', routing_key='task.#'),
                 Queue('frontend_tasks', routing_key='frontend.#'),
                )

CELERY_DEFAULT_ROUTING_KEY = 'task.default'
CELERY_DEFAULT_EXCHANGE = 'tasks'

Celery version: 3.1.13

RabbitMQ version: 3.3.4-1

kombu version: 3.0.21

amqp version: 1.4.5

Invocation of Celery worker:

python manage.py celery worker -f c:\logs\celery.log -E -l info --settings=[proj].settings.production

UPDATE: After setting the prefetch multiplier to 1, I can still see the worker even when they have four tasks. However, I cannot see any pending tasks on the worker because they have not been fetched from the server. Does anyone know if there is a way I can keep the worker (appearing) online and still see pending tasks in the queue? Setting prefetch multiplier even to 2 makes the worker drop from my inspect/flower whenever they have four or more tasks. Even setting prefetch multiplier to 2 and concurrency to 1 or 2 (down from 4) causes the same "disappearing worker" issue.

Thanks in advance for the help,

Steve

Steve Cobb Steve Cobb · Accepted Answer · 2014-08-05T15:52:50

I found a solution that fixed this issue. I changed my workers to threaded as opposed to prefork by submitting the -P threads argument to the invocation:

python manage.py celery worker -f c:\logs\celery.log -E -P threads -l info --settings=[proj].settings.production

I was then able to correctly inspect the workers even when they were running at their concurrency value. I did have to ensure that all of my tasks were threadsafe (shutil was the major offender, and a simple lock fixed it).

I hope this helps.

Steve

Celery inspect shows worker offline when Concurrency value reached

1 Answers