daemonizing celery process celeryd-multi not found

Question

I'm trying to daemonize the celery process for django running inside a virtualenv. I copied the celeryd file from https://github.com/celery/celery/tree/master/extra/generic-init.d to /etc/init.d/

I then created a config file with the contents at http://ask.github.io/celery/cookbook/daemonizing.html#example-django-configuration-using-virtualenv and saved it as /etc/default/celeryd

This is my /etc/default/celeryd:

# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Where to chdir at start.
CELERYD_CHDIR="/home/manu/location/to/project/"

# Python interpreter from environment.
ENV_PYTHON="/home/manu/.virtualenvs/project_env/bin/python"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI= "$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"

# Extra arguments to celeryd
CELERYD_OPTS="--verbose --fake --time-limit=300 --concurrency=8"

# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="project.settings.production"

When I run:

sudo sh -x /etc/init.d/celeryd start

It fails with the following error:

celeryd-multi start w1 --uid=celery --gid=celery --workdir=/home/manu/location/to/project/ --pidfile=/var/run/celery/%n.pid --logfile=/var/log/celery/%n.log --loglevel=INFO --cmd=-m celery.bin.celeryd_detach --verbose --fake --time-limit=300 --concurrency=8 /etc/init.d/celeryd: 140: /etc/init.d/celeryd: celeryd-multi: not found

I can see that it's unable to find celeryd_multi. Strangely, running

/path/to/project_env/bin/python /path/to/manage.py celeryd_multi

is showing that celeryd-multi is available.

[Update - Correcting my start command with @Daniel Roseman's input]

Now, when I run:

sh -x /etc/init.d/celeryd start

This is the output I see:

/path/to/.virtualenvs/virtenv/bin/python /path/to/project//manage.py celeryd_detach --time-limit=300 --concurrency=8 --gid=celery --broker=amqp://:@localhost:5672// -n w1.ubuntu-12-10 --logfile=/var/log/celery/w1.log --loglevel=INFO --uid=celery --pidfile=/var/run/celery/w1.pid --workdir=/path/to/project/

OK

+ sleep 5

+ exit 0

What am I missing here? Please help!

sontek sontek · Accepted Answer · 2013-06-05T05:10:26

Your init.d script is referencing the system python package, not the virtualenv python. So when you are running celery inside it, you are referencing the one installed in your system site packages.

Add a reference to the virtualenv python path before calling celery and everything should work just fine.

That being said, it is usually better to use init.d to run supervisord and then have supervisord spin up all your other processes like celery and your web application.

daemonizing celery process celeryd-multi not found

2 Answers