Celery kills processes

Question

I use celery 3.1.23.

Task code is very simple:

@app.task(bind=True, max_retries=None, default_retry_delay=settings.CELERY_RECONNECT_TIME)
def analize_text(self, **kwargs):
    print 'test'
    return 1

I launch celery this command:

celery -A tasks worker --loglevel=info --concurrency=4 -n analize_text -Q analize.analize_text -Ofair

So, i use 4 CPU. The problem is that celery periodically kills workers and starts new ones instead. New worker has new number in log and new PID. Here is my log https://dpaste.de/N1Vk

Why does it do it? Is this bug?

Answer 1

It depends on CELERY_MAX_TASKS_PER_CHILD i.e. number of tasks a worker shall complete before being recycled, default is no limit.

http://docs.celeryproject.org/en/latest/configuration.html#std:setting-CELERYD_MAX_TASKS_PER_CHILD

So,your celery configuration might be having this limit set as some low number, hence your workers are getting recycled. You can see more information in celery stats.