django 1.3 logging for debugging in development environment

Question

I'm using the logging framework in django 1.3 as described here https://docs.djangoproject.com/en/1.3/topics/logging/

In my settings.py file for my DEV environment, i basically have custom formatter, logger and handler defined so that i can log debug output to a file.

So now in my code i've got logger.debug() statements all over the place.

My questions are:

1. When i deploy in production do the logger.debug statements have any (signficant) effect on performance? (i.e. do i need to take them all out?)
2. What is the best way to change the settings.py file so that it no longer captures to file? Currently i'm just changing the handler to django.utils.log.NullHandler

Answer 1

1. All those debug statements will have a very small effect on overall performance - the problems in performance will show up if you are trying to write all those log messages to a file somewhere when under heavy load. Writing a lot of stuff to files can be CPU and IO intensive. This wont be as much of an issue if you have your log level set low (WARNING) since that should not result in a lot of messages logged. Keeping the debug statements in there will be helpful in the future when you want to debug/enhance that code.
2. Using the null handler if fine if you don't what the messages logged to file or std out. Alternatively you could turn the logging level down to WARNING so that only important messages come through.

Answer 2

You should still have quite a bit of logging in your production config, as that is where it's going to be hardest to find out what a problem is, so make sure you don't strip too much out.

In terms of production/development settings, I always have this at the bottom of my settings.py:

try:
    from local_settings import *
except ImportError:
    pass

Then I have a local_settings.py that I don't add to source control and can override settings in. That way you can have separate production/dev settings really easily.

I took the idea from this blog and have used it on lots of projects now - very handy. The blog expands on the link and shows how you can add development and production settings to source control, then use local_settings to include one or the other.

Answer 3

For production, you should always enable debugging - as others have mentioned this is where you need it the most.

In production we use sentry for application level logging and splunk to gather system logs for easy searching.

The overall value you get from proper debugging statements negates any possible effect they would have on performance. Think of it another way - if you didn't have debugging, if your system was not performing well you wouldn't be able to know where it was choking.

Debugging statements are the last place I would look to increase performance. You would get more bang for your buck if you implemented memcache or redis; or simply optimized your queries (as this is where the majority of the performance problems start).