Does Celery cache tasks? How can I reload changes in a dev environment?

Question

When using Celery, Docker, RabbitMQ and Django (with django-celery-results and django-celery-beat) I am able to load a simple task following the tutorial.

However, when I make changes to the task and reload the server (docker-compose down, docker-compose up) the changes are not reflected. Does Celery cache tasks somewhere / how to I reload them when in a dev environment? The tutorial sets CELERY_CACHE_BACKEND = 'django-cache' but I would assume this is destroyed by docker-compose down?

For example, removing the task from both tasks.py and CELERY_BEAT_SCHEDULE in settings.py does not prevent it loading when the server is restarted.

celery.py:

from __future__ import absolute_import, unicode_literals

import os
from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')
app = Celery('proj')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

proj/__ init __ .py:

from __future__ import absolute_import, unicode_literals
from .celery import app as celery_app
__all__ = ('celery_app',)

tasks.py:

from __future__ import absolute_import, unicode_literals
from celery import shared_task

@shared_task
def hello_world():
   return print('hello world!')

settings.py:

CELERY_BROKER_URL = 'pyamqp://rabbitmq:5672'
CELERY_RESULT_BACKEND = 'django-db'
CELERY_CACHE_BACKEND = 'django-cache'
CELERY_BEAT_SCHEDULER = 'django_celery_beat.schedulers:DatabaseScheduler'
CELERY_BEAT_SCHEDULE = {
    'hello':
        {
            'task': 'proj.tasks.hello_world',
            'schedule':
                crontab()  # execute every minute
        }
}

iklinac iklinac · Accepted Answer · 2020-04-18T17:22:10

There is no autoreload function in celery ( it was removed for version 4 and it was experimental)

The --autoreload feature has been removed.

This was an experimental feature, and not covered by our deprecation timeline guarantee. The flag is removed completely so the worker will crash at startup when present. Luckily this flag isn’t used in production systems.

You could write your own solution to watch for file changes using watchdog or something else to restart worker, or just restart container/ rerun worker

Does Celery cache tasks? How can I reload changes in a dev environment?

1 Answers