Cannot setup Celery as daemon on server

Question

I cannot setup Celery as daemon on server (django 1.6.11, celery 3.1, Ubuntu 14.04) Tried lot of options, can anyone place full setting of working configuration to run celery as daemon?

I am very disappointed from official docs http://docs.celeryproject.org/en/latest/tutorials/daemonizing.html#generic-init-scripts - none of this working, no full step-by-step tutorial. Zero (!!!) videos on youtube on how to setup daemon.

Now i able to run celery simple by celery worker -A engine -l info -E tasks from django are executed successfully.

I have done configs:

/etc/defaults/celery

    # Name of nodes to start
# here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Absolute path to "manage.py"
CELERY_BIN="/var/www/engine/manage.py"

# How to call manage.py
CELERYD_MULTI="celery multi"

# Extra command-line arguments to the worker
CELERYD_OPTS="--time-limit=300 --concurrency=2"

# %N will be replaced with the first part of the nodename.
CELERYD_LOG_FILE="/var/log/celery/%N.log"
CELERYD_PID_FILE="/var/run/celery/%N.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="root"
CELERYD_GROUP="root"

/etc/init.d/celeryd
got from https://github.com/celery/celery/blob/3.1/extra/generic-init.d/celeryd without changes

Now, when i go to console and run: cd /etc/init.d celery multi start w1

i see output:

celery multi v3.1.11 (Cipater)
> Starting nodes...
        > w1@engine: OK

So, no errors! Tasks are not invoked and i cannot figure out whats wrong.

Mikhail Pyrev Mikhail Pyrev · Accepted Answer · 2016-01-28T06:14:31

I would suggest to use Supervisor. It's better way than init scripts, because you can run multiple Celery instances for different projects on one server. Example config for Supervisor you can find in Celery repo or fully working example from my project:

# /etc/supervisor/conf.d/celery.conf
[program:celery]
command=/home/newspos/.virtualenvs/newspos/bin/celery worker -A newspos --loglevel=INFO
user=newspos
environment=DJANGO_SETTINGS_MODULE="newspos.settings"
directory=/home/newspos/projects/newspos/
autostart=true
autorestart=true
stopwaitsecs = 600
killasgroup=true
startsecs=10
stdout_logfile=/var/log/celery/newspos-celeryd.log
stderr_logfile=/var/log/celery/newspos-celeryd.log

Cannot setup Celery as daemon on server

1 Answers