Celery logging to file from inside the Django modules

Question

I have the following in my tasks.py

from celery.utils.log import get_task_logger
logger = get_task_logger("celery.task")

I have setup logging for celery.task in my settings.py and all the log from my tasks.py file are properly logged to file.

I have Django modules. These modules can be called directly from Django or by a Celery task. What I want is, if the module is called by Django, then logs should go to Django log file. If the module is called by a task, the log should go to celery task logger.

Example:

# tasks.py

from app.foo import MyFoo

@task(name="bar")
def boo():
    MyFoo.someFoo()

# app.foo.py

log = logging.getLogger(__name__)

I want the log messages inside MyFoo to go to celery log when run by a worker task.

Any ideas?

Mikko Ohtamaa Mikko Ohtamaa · Accepted Answer · 2013-12-20T08:21:14

You should be able to configure loggers separately for Django process and Celery process, as if by definition they run in separate processes.

Actually I am surprised that the log output does not go to a separate log files; thus maybe it would make sense to you expose what kind of logging configurations you have already in-place.

Where Python logging output goes is defined by logging handlers.

Configuring logger for Django: https://docs.djangoproject.com/en/dev/topics/logging/

For Celery, there doesn't seem to be straightforward way to override logging settings. They seem to give a celery.signals.setup_logging where you could set a breakpoint, use logging API to reconfigure logging handlers to go to a separate log file.

http://docs.celeryproject.org/en/latest/configuration.html#logging

Alternatively, you can just pull out different logger object on a task level. When you execute a task you know whether it is executed from Django (eager) or Celery.

I have never done this myself, but apparently Celery tasks expose the task running context as self.request parameter (nothing to do with Python classes).

http://docs.celeryproject.org/en/latest/userguide/tasks.html#context

So at the beginning of your task function you could switch between the loggers:

# Define web_worker_logger
# Define task_logger

@task(name='example')
def moobar():
    logger = web_worker_logger if self.request.is_eager else task_logger

Celery logging to file from inside the Django modules

1 Answers