Python logging multiple files using the same logger

15
votes

This is my scenario: I want to log my_module's activity. This needs to be done, depending on the method executed (let's say, INPUT and OUTPUT), to two different files.

So I have two Handlers, each one point to a different file (my_in_.log & my_out_.log), with the same log level. I would like to know if I can use the same logger to achieve this or I have to define two loggers. My config is:

[loggers]
keys=root, my_log

[handlers]
keys=my_in_hand, my_out_hand

[formatters]
keys=generic_form


...


[logger_my_log]
level=NOTSET
handlers=my_in_hand, my_out_hand
qualname=ws_log

[handler_my_in_hand]
class=handlers.TimeRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('my_in_.log', 'h', 1, 0, None, False, True)

[handler_my_out_hand]
class=handlers.TimeRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('my_out_.log', 'h', 1, 0, None, False, True)

Do I have to define a logger per handler/destination (because I want to log different information in different files)? Is there a way to indicate to the logger which handler will do this? I mean, I have two handlers for one logger, then choose only one handler to log one method.

3
pythonloggingconfig

3 Answers

9
votes

You should instantiate an Handler for each destination you want to send your log to, then add the 2 handlers to your logger. The following should work (didn't test it though):

logger = logging.getLogger()
handler1 = logging.TimedRotatingFileHandler()
handler2 = logging.TimedRotatingFileHandler()
logger.addHandler(handler1)
logger.addHandler(handler2)

Of course add all your configuration and formatting options you may need. Basically it is just to show you that when you instantiate the logging handler you can add it to the logger. From that moment on, your log records will be emitted to every handler added to the logger.

7
votes

what you want is to

  1. create 2 NON ROOT loggers.
  2. make handler for each one, point to different file

  3. add handler to appropriate logger

    logger1 = logging.getLogger('general_logger')
logger2 = logging.getLogger('some_other_logger')

log_handler1 = logging.handlers.RotatingFileHandler(file_1, *args)
log_handler2 = logging.handlers.RotatingFileHandler(file_2, *args)

logger1.addHandler(log_handler1)
logger2.addHandler(log_handler2)

then

    logger1.info("this will be logged to file_1 ")
    logger2.info("this will be logged to file_2 ")

Please note that if you create a ROOT logger and a different logger, root logger will log everything that this different controller is trying to log.

In other words, if

    root_logger = logging.getLogger()
    logger2 = logging.getLogger('some_other_logger')

    root_log_handler = logging.handlers.RotatingFileHandler(file_1, *args)
    log_handler2 = logging.handlers.RotatingFileHandler(file_2, *args)

    root_logger.addHandler(root_log_handler)
    logger2.addHandler(log_handler2)

then

    root_logger.info("this will be logged to file_1 ")
    logger2.info("this will be logged to file_1 AND file_2 ")
6
votes

Finally I decided to define two loggers, because:

  • They are for different purposses. In my case, one logs input request to a web service, and the other one logs the response. And they use different files to it

  • I'm using a logging config file, in a frontal web service. Adding/removing handlers before logging messages is not the right approach, as @mike said. Thx to @drekyn too!

Here is my logging config file, just for reference if anyone is interested in:

[loggers]
keys=root, ws_in_log, ws_out_log

[handlers]
keys=consoleHandler, ws_in_hand, ws_out_hand

[formatters]
keys=generic_form

[logger_root]
handlers=consoleHandler
level=NOTSET

[logger_ws_in_log]
level=NOTSET
handlers=ws_in_hand
qualname=ws_in_log

[logger_ws_out_log]
level=NOTSET
handlers=ws_out_hand
qualname=ws_out_log

[handler_ws_in_hand]
class=logging.handlers.TimedRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('/path/ws_in_.log', 'h', 1, 0, None, False, True)

[handler_ws_out_hand]
class=logging.handlers.TimedRotatingFileHandler
level=NOTSET
formatter=generic_form
args=('/path/em/ws_out_.log', 'h', 1, 0, None, False, True)

[handler_consoleHandler]
class=StreamHandler
level=DEBUG
formatter=generic_form
args=(sys.stdout,)

[formatter_generic_form]
format='%(asctime)s - %(levelname)s - %(message)s'
datefmt='%Y-%m-%d %H:%M:%S'
class=

See you!