python - How do you log server errors on django sites

Question

So, when playing with the development I can just set settings.DEBUG to True and if an error occures I can see it nicely formatted, with good stack trace and request information.

But on kind of production site I'd rather use DEBUG=False and show visitors some standard error 500 page with information that I'm working on fixing this bug at this moment ;)
At the same time I'd like to have some way of logging all those information (stack trace and request info) to a file on my server - so I can just output it to my console and watch errors scroll, email the log to me every hour or something like this.

What logging solutions would you recomend for a django-site, that would meet those simple requirements? I have the application running as fcgi server and I'm using apache web server as frontend (although thinking of going to lighttpd).

Answer 1

Well, when DEBUG = False, Django will automatically mail a full traceback of any error to each person listed in the ADMINS setting, which gets you notifications pretty much for free. If you'd like more fine-grained control, you can write and add to your settings a middleware class which defines a method named process_exception(), which will have access to the exception that was raised:

http://docs.djangoproject.com/en/dev/topics/http/middleware/#process-exception

Your process_exception() method can then perform whatever type of logging you'd like: writing to console, writing to a file, etc., etc.

Edit: though it's a bit less useful, you can also listen for the got_request_exception signal, which will be sent whenever an exception is encountered during request processing:

http://docs.djangoproject.com/en/dev/ref/signals/#got-request-exception

This does not give you access to the exception object, however, so the middleware method is much easier to work with.

Answer 2

Django Sentry is a good way to go, as already mentioned, but there is a bit of work involved in setting it up properly (as a separate website). If you just want to log everything to a simple text file here's the logging configuration to put in your settings.py

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'handlers': {
        # Include the default Django email handler for errors
        # This is what you'd get without configuring logging at all.
        'mail_admins': {
            'class': 'django.utils.log.AdminEmailHandler',
            'level': 'ERROR',
             # But the emails are plain text by default - HTML is nicer
            'include_html': True,
        },
        # Log to a text file that can be rotated by logrotate
        'logfile': {
            'class': 'logging.handlers.WatchedFileHandler',
            'filename': '/var/log/django/myapp.log'
        },
    },
    'loggers': {
        # Again, default Django configuration to email unhandled exceptions
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        # Might as well log any errors anywhere else in Django
        'django': {
            'handlers': ['logfile'],
            'level': 'ERROR',
            'propagate': False,
        },
        # Your own app - this assumes all your logger names start with "myapp."
        'myapp': {
            'handlers': ['logfile'],
            'level': 'WARNING', # Or maybe INFO or DEBUG
            'propagate': False
        },
    },
}

Answer 3

django-db-log, mentioned in another answer, has been replaced with:

https://github.com/dcramer/django-sentry

Answer 4

Obviously James is correct, but if you wanted to log exceptions in a datastore, there are a few open source solutions already available:

1) CrashLog is a good choice: http://code.google.com/p/django-crashlog/

2) Db-Log is a good choice as well: http://code.google.com/p/django-db-log/

What is the difference between the two? Almost nothing that I can see, so either one will suffice.

I've used both and they work well.

Answer 5

Some time has passed since EMP's most helpful code submission. I just now implemented it, and while thrashing around with some manage.py option, to try to chase down a bug, I got a deprecation warning to the effect that with my current version of Django (1.5.?) a require_debug_false filter is now needed for the mail_admins handler.

Here is the revised code:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
         'require_debug_false': {
             '()': 'django.utils.log.RequireDebugFalse'
         }
     },
    'handlers': {
        # Include the default Django email handler for errors
        # This is what you'd get without configuring logging at all.
        'mail_admins': {
            'class': 'django.utils.log.AdminEmailHandler',
            'level': 'ERROR',
            'filters': ['require_debug_false'],
             # But the emails are plain text by default - HTML is nicer
            'include_html': True,
        },
        # Log to a text file that can be rotated by logrotate
        'logfile': {
            'class': 'logging.handlers.WatchedFileHandler',
            'filename': '/home/username/public_html/djangoprojectname/logfilename.log'
        },
    },
    'loggers': {
        # Again, default Django configuration to email unhandled exceptions
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        # Might as well log any errors anywhere else in Django
        'django': {
            'handlers': ['logfile'],
            'level': 'ERROR',
            'propagate': False,
        },
        # Your own app - this assumes all your logger names start with "myapp."
        'myapp': {
            'handlers': ['logfile'],
            'level': 'DEBUG', # Or maybe INFO or WARNING
            'propagate': False
        },
    },
}

Answer 6

I just had an annoying problem with my fcgi script. It occurred before django even started. The lack of logging is sooo painful. Anyway, redirecting stderr to a file as the very first thing helped a lot:

#!/home/user/env/bin/python
sys.stderr = open('/home/user/fcgi_errors', 'a')