Python: Log to multiple log files

Question

Currently I have everything getting logged to one logfile but I want to separate it out to multiple log files. I look at the logging in python documentation but they don't discuss about this.

log_format = "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
logging.basicConfig(filename=(os.path.join(OUT_DIR, + '-user.log')),
            format=log_format, level=logging.INFO, datefmt='%Y-%m-%d %H:%M:%S')

Currently this is how I do the logging. what I want to do have different type of errors or information get log into different log files. At the moment when I do logging.info('Logging IN') and logging.error('unable to login') will go to same logfile. I want to seperate them. Do I need to create another logging object to support the logging into another file?

Answer 1

What you /could/ do (I haven't dug into the logging module too much so there may be a better way to do this) is maybe use a stream rather than a file object:

In [1]: class LogHandler(object):
   ...:     def write(self, msg):
   ...:         print 'a :%s' % msg
   ...:         print 'b :%s' % msg
   ...:         

In [3]: import logging
In [4]: logging.basicConfig(stream=LogHandler())
In [5]: logging.critical('foo')
a :CRITICAL:root:foo
b :CRITICAL:root:foo

In [6]: logging.warn('bar')
a :WARNING:root:bar
b :WARNING:root:bar

Edit with further handling:

Assuming your log files already exist, you could do something like this:

import logging

class LogHandler(object):
    format = '%(levelname)s %(message)s'
    files = { 
        'ERROR': 'error.log',
        'CRITICAL': 'error.log',
        'WARN': 'warn.log',
    }   
    def write(self, msg):
        type_ = msg[:msg.index(' ')] 
        with open(self.files.get(type_, 'log.log'), 'r+') as f:
            f.write(msg)

logging.basicConfig(format=LogHandler.format, stream=LogHandler())
logging.critical('foo')

This would allow you to split your logging into various files based on conditions in your log messages. If what you're looking for isn't found, it simply defaults to log.log.

Answer 2

I created this solution from docs.python.org/2/howto/logging-cookbook.html

Simply create two logging file handlers, assign their logging level and add them to your logger.

import os
import logging

current_path = os.path.dirname(os.path.realpath(__file__))

logger = logging.getLogger('simple_example')
logger.setLevel(logging.DEBUG)

#to log debug messages                               
debug_log = logging.FileHandler(os.path.join(current_path, 'debug.log'))
debug_log.setLevel(logging.DEBUG)

#to log errors messages
error_log = logging.FileHandler(os.path.join(current_path, 'error.log'))
error_log.setLevel(logging.ERROR)

logger.addHandler(debug_log)
logger.addHandler(error_log)

logger.debug('This message should go in the debug log')
logger.info('and so should this message')
logger.warning('and this message')
logger.error('This message should go in both the debug log and the error log')