Celery with Django and MongoDB (mongoengine)

Question

1) I am trying to build an application using Celery (with RabbitMQ as broker) and Django - using MongoDB (mongoengine) as the database for the model. So the requests received by the web server will be transformed in tasks and queued with the help of Celery to be executed by the workers.

I followed the following tutorials:

http://docs.celeryproject.org/en/master/django/first-steps-with-django.html#configuring-your-django-project-to-use-celery

and

https://mongoengine-odm.readthedocs.org/en/latest/django.html

but I still get the following error:

ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value.

As mentioned in both tutorials, settings.DATABASES should be commented and instead of it there should be only

mongoengine.connect('myDB')

and yet the error is exactly about not having the DATABASES configured.

(Apart from that, I have not configure any results-backend for Celery.)

Can anybody help me with an advice as to what I have to set and where?

2) And another question is: in the projects involving only Celery there is always a Celery instance. But in the tutorials about building web applications with Django and Celery I haven't seen any mention of this. Do I have to explicitly instantiate Celery or is this done somewhere else by default?

Answer 1

1) In case anybody is interested in the answer, I finally managed to get it working, but I am not sure if I understood correctly what happened.

So apparently the problem was that I haven't set the result backend for Celery. I got rid of the error as soon as I put the following line in settings.py:

CELERY_RESULT_BACKEND = "amqp"

2) My project (I am using djcelery) is working without me explicitly instantiating Celery. I assume this is done somewhere in the back by the framework.

Answer 2

What happens is that Celery attempts to use Django's default database (the one defined in settings.DATABASES) as the Celery Result database, but to use mongoengine as your primary Django database you have to bypass settings.DATABASES.

So just make sure you define both BROKER_URL and CELERY_RESULT_BACKEND properly so that Celery doesn't try to consult settings.DATABASES. I guess you want them to be the same, but you could chose to have them separate.

BROKER_URL = "amqp://guest:guest@localhost:5672//"
CELERY_RESULT_BACKEND = "amqp"

For other backends, consult this.

---

Part 2 of your question.

Do you have CELERY_ALWAYS_EAGER = True in your settings.py? That is how, typically, the Celery process doesn't need to be launched separately. But do not use this in production. See this question.