Django - Celery - supervisord logging configuration

Question

i have a django project with celery task (running inside), but i have a question about logging, what i have done is:

Task logging get_task_logger :

from celery.utils.log import get_task_logger
logger = get_task_logger('celery')

Django logging:

LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'root': {
    'level': 'DEBUG',
    'handlers': ['sentry', 'file'],
},
'formatters': {
    'verbose': {
        'format': '%(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'
    },
},
'handlers': {
    'console': {
        'level': 'DEBUG',
        'class': 'logging.StreamHandler',
        'formatter': 'verbose',
    },
    'celery': {
        'level': 'INFO',
        'class': 'logging.StreamHandler',
        'stream': sys.stdout,
    },
 ........
},
'loggers': {
    'celery': {
        'handlers': ['celery'],
        'level': 'INFO',
    },
......
},

supervisord celery config:

[program:celery] *command=/usr/local/bin/celery worker -A app --autoscale=20,10 -E -l INFO -Ofair
directory=/xxx/celerydir
numprocs=1
stdout_logfile=/xxx/logs/celery_worker.log
stderr_logfile=/xxx/logs/celery_worker_err.log
autostart=true
autorestart=true
startsecs=10
stopwaitsecs=600
stopasgroup=true
priority=998

My problem is as you can see, inside Django -> celery handler i've specified "'stream': sys.stdout", but when i start celery with supervisord i will see both logs celery_worker.log and celery_worker_err.log that store all logs for all level : INFO, WARNING, ERROR... why?

How can i configure logger for celery to redirect all celery log only to stdout and let supervisord store level info only on celery_worker.log ?

thanks

marcinkuzminski marcinkuzminski · Accepted Answer · 2015-10-16T10:01:52

I think it's because by default celery reports things to stderr instead of stdout, you can use supervisord redirect_stderr = true flag to combine two files into one if that helps.

There's a settings in celery that let's you specify a log-file if log-file is not specified it uses stderr.

Django - Celery - supervisord logging configuration

1 Answers