django celery daemon doesn't work

Question

Here's my celery app config:

from __future__ import absolute_import

from celery import Celery

import os

from django.conf import settings

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tshirtmafia.settings')

app = Celery('tshirtmafia')
app.conf.update(
    CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)

app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

settings.py:

INSTALLED_APPS:

'kombu.transport.django',
'djcelery',

also:

BROKER_URL = 'django://'

Here's my task:

@shared_task
def test():
    send_mail('nesamone bus', 'Files have been successfully generated.', 'marijus.merkevicius@gmail.com',
    ['marijus.merkevicius@gmail.com'], fail_silently=False)

Now when I run locally python manage.py celeryd locally and then run test.delay() from shell locally it works.

Now I'm trying to deploy my app. When with the exact same configuration I try to open python manage.py celeryd and in other window I open shell and run test task, it doesn't work.

I've also tried to setup background daemon like this:

/etc/default/celeryd configuration:

# Name of nodes to start, here we have a single node
CELERYD_NODES="w1"
# or we could have three nodes:
#CELERYD_NODES="w1 w2 w3"

# Where to chdir at start. (CATMAID Django project dir.)
CELERYD_CHDIR="/home/tshirtnation/"

# Python interpreter from environment. (in CATMAID Django dir)
ENV_PYTHON="/usr/bin/python"

# How to call "manage.py celeryd_multi"
CELERYD_MULTI="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryd_multi"

# How to call "manage.py celeryctl"
CELERYCTL="$ENV_PYTHON $CELERYD_CHDIR/manage.py celeryctl"

# Extra arguments to celeryd
CELERYD_OPTS="--time-limit=300 --concurrency=1"

# Name of the celery config module.
CELERY_CONFIG_MODULE="celeryconfig"

# %n will be replaced with the nodename.
CELERYD_LOG_FILE="/var/log/celery/%n.log"
CELERYD_PID_FILE="/var/run/celery/%n.pid"

# Workers should run as an unprivileged user.
CELERYD_USER="celery"
CELERYD_GROUP="celery"

# Name of the projects settings module.
export DJANGO_SETTINGS_MODULE="settings"

And I use default celery /etc/init.d/celeryd script.

So basically it seems like celeryd starts but doesn't work. No idea how to debug this and what might be wrong.

Let me know if you need anything else

igolkotek igolkotek · Accepted Answer · 2015-03-31T06:07:09

Celery turned to be a very capricious child in Django robust system as for me. There are too little initial data for understanding the reason of your problems. The most usual reason of Celery daemon fail is file system permissions. But to clarify the reason I'd try:

Start celery from a command line by the user-owner of django project:

celery -A proj worker -l info

If it works OK, go further

Start celery in a verbal mode as a root user just like daemon to be:

sudo sh -x /etc/init.d/celeryd start

This will show most of the problems with the daemon script - celery user and group used, but not all, unfortunately: permission fails are not visible.

My little remark. Usually Celery is started by own celery user, and the django project by another one. After long fighting celery and system, I refused from celery user, and owned celery process by the django project user.

And .. do not forget to start once

update-rc.d celerybeat defaults
update-rc.d celeryd defaults

this is for Ubuntu daemon start, sure.

Good luck

django celery daemon doesn't work

1 Answers