Unable to email in Django via Celery worker when using System V init script

Question

I have a Django 1.6.2 application and I'm using Celery 3.1.7 and RabbitMQ 2.8.4 to handle asynchronous tasks. My problem is that I'm unable to send an email using Django's send_mail function if the email is sent via a Celery worker queue. When I execute this send_email task...

# apps.photos.tasks.py
from __future__ import absolute_import
from django.core.mail import send_mail
from conf.celeryapp import app

@app.task
def send_email():
    send_mail('Test Email',
        'This is a test email.',
        'me@example.com',
        ['me2@example.com'],
        fail_silently=False)

via this command in my Python interpreter...

>>> from apps.photos.tasks import send_email
>>> result = send_email.delay()

I get the following error:

[2015-04-24 15:13:48,371: ERROR/MainProcess] Task apps.photos.tasks.send_email[db303245-f36c-4ae2-9d72-86d2bbbbd358] raised unexpected: ImproperlyConfigured('Requested setting EMAIL_BACKEND, but settings are not configured. You must either define the environment variable DJANGO_SETTINGS_MODULE or call settings.configure() before accessing settings.',)
Traceback (most recent call last):
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
    return self.run(*args, **kwargs)
  File "apps/photos/tasks.py", line 34, in send_email
    fail_silently=False)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 48, in send_mail
    fail_silently=fail_silently)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/core/mail/__init__.py", line 29, in get_connection
    klass = import_by_path(backend or settings.EMAIL_BACKEND)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 54, in __getattr__
    self._setup(name)
  File "/home/myproj/venv/myproj/local/lib/python2.7/site-packages/django/conf/__init__.py", line 47, in _setup
    % (desc, ENVIRONMENT_VARIABLE))

The environment variable DJANGO_SETTINGS_MODULE is set in my virtual environment.

This only happens if I start my workers via this celery daemon init script:

# /etc/default/celeryd
CELERYD_NODES="worker1 worker2 worker3"
CELERY_BIN="/home/myproj/venv/myproj/bin/celery"
CELERY_APP="conf.celeryapp:app"
CELERYD_CHDIR="/www/myproj"
CELERYD_OPTS="--time-limit=300 --concurrency=8 --config=celeryconfig -Q:1 default -Q:2 photos -Q:3 mail"
CELERYD_USER="celery"
CELERYD_GROUP="celery"

(My Celery init script /etc/init.d/celeryd is the standard one pointed to by the Celery docs.)

If I start the worker and queue from the command line, the email is sent without any error:

celery worker -A conf.celeryapp:app -n worker3 -Q mail -l info

Here's my celeryconfig file:

# /www/myproj/conf/celeryconfig.py
from kombu import Queue, Exchange

BROKER_URL = 'amqp://'
CELERY_RESULT_BACKEND = 'amqp://'
CELERY_DEFAULT_QUEUE = 'default'
CELERY_QUEUES = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('photos', Exchange('photos'), routing_key='photos'),
    Queue('mail', Exchange('mail'), routing_key='mail'),
)
CELERY_ROUTES = (
    {'apps.photos.tasks.hello': {'queue': 'default', 'routing_key': 'default'}},
    {'apps.photos.tasks.add': {'queue': 'photos', 'routing_key': 'photos'}},
    {'apps.photos.tasks.send_email': {'queue': 'mail', 'routing_key': 'mail'}},
)

This file initializes the Celery app:

# /www/myproj/conf/celeryapp.py
from celery import Celery

#import os
#os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'conf.settings.prod')

app = Celery('celeryapp')
app.config_from_object('conf.celeryconfig')
app.autodiscover_tasks(['apps.photos'])

These settings configure my Django mail to use a 3rd party email service:

# /www/myproj/conf/settings/base.py
...
EMAIL_HOST = 'smtp.sendgrid.net'
EMAIL_HOST_USER = get_env_variable('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD = get_env_variable('EMAIL_HOST_PASSWORD')
EMAIL_PORT = 587
EMAIL_USE_TLS = True
SITE_EMAIL = 'admin@example.com'
RECIPIENTS = ['me@example.com', ]
FAIL_SILENTLY = False
...

I tried uncommenting the "os.environ.setdefault" command in celeryapp.py thinking that Celery needs explicit directions to the settings file but when I do that and try to start celery via my celeryd script with the command "sudo service celeryd start", the workers won't start up at all and yet Celery doesn't display any errors, even when I set the loglevel to "debug".

I also tried explicitly setting the EMAIL_BACKEND variable in my settings file as well as the send_email function itself...

settings.EMAIL_BACKEND = 'django.core.mail.backends.smtp.EmailBackend'

but that didn't resolve the error.

Additionally, I added the following to my /etc/default/celeryd file to no avail:

# /etc/default/celeryd
...
export DJANGO_SETTINGS_MODULE="conf.settings.prod"
...

Can anyone see what I'm doing wrong?

Thanks!

pbhowmick pbhowmick · Accepted Answer · 2015-04-27T15:42:48

It's because your worker is not a Django app at all, only your dispatcher is. The Django config for SMTP is really meant to send an email directly from the context of a a Django view or a Django signal handler. Actually, what you need to do is use smtplib from your worker process to send the email and DO NOT try to read the django config file. Instead, the worker process should read the SMTP config params from something like a JSON file

Unable to email in Django via Celery worker when using System V init script

1 Answers