Using one logger with multiple file handlers that are almost same

Question

I have a service that is responsible for 2 flows: A and B. Those flows are connected but I want to log each flow in a different log file.

I thought using the following config file:

[loggers]
keys=root,A,B

[handlers]
keys=file_handler

[formatters]
keys=full

[logger_A]
level=INFO
handlers=fileHandler

[logger_B]
level=INFO
handlers=fileHandler

[logger_root]
level=INFO
handlers=fileHandler

[handler_fileHandler]
class=FileHandler
level=INFO
formatter=full
args=('%(logfilename)s','%(permissions)s','%(encoding)s',)

[formatter_full]
format=[%(asctime)s] [%(levelname)s] --- [%(funcName)s] [P%(process)d][%(threadName)s - %(thread)d]- %(message)s
datefmt='%Y-%m-%d %H:%M:%S'

From what I understood there isn't any way to send those args to the fileHandler dynamically during runtime, right? The handler has the same attributes for all loggers the difference is the destination file that the logger writes to.

I thought about configuring a function that will get as an input the logger name and the destination file and will return me the logger - it is something that was suggested in Logging to two files with different settings:

import logging
def getLoggerToFile(loggerName,filePath,fileMode,fileHandler=None,level=logging.INFO):
    logger = logging.getLogger(loggerName)
    if len(logger.handlers) > 0:
         return logger
    if(fileHandler is None):
        formatter = logging.Formatter('[%(asctime)s] [%(levelname)s] --- [%(funcName)s] '
                                      '[P%(process)d][%(threadName)s - %(thread)d]- %(message)s')
        fileHandler = logging.FileHandler(filePath, mode=fileMode)
        fileHandler.setFormatter(formatter)

    logger.setLevel(level)
    logger.addHandler(fileHandler)
    return logger

Wanted to hear of any other suggestions and to ask a question:

If I'll use the function, and I'll call the function twice with the same logger name but with different file path, the logger won't be able to write simultaneously to 2 different files, right? So it's the same as creating a config file with 2 loggers and a handler/formatter for each logger.

Can you take a look at logbook? It's just waaay easier to configure than the POS of builtin logging lib... Excuse my rant ;) — Christian Sauer
I will check it out, still wanna find a solution for this case ;) — JeyJ
Do you just want logger A and B to write their logs to different files? In that case just create two filehandlers. Or have I misunderstood your question? — blues
I want to log different data to two different files with the same logger. What u suggested is basically the function that I implemented — JeyJ
Your question is still not entirely clear to me, but have you looked at log filters? You can create a logger with two handlers, with a filter on each handler so that it only handles the data you want — blues

blues blues · Accepted Answer · 2019-09-03T15:09:20

It is possible to do this with a Filter and two handlers. Create a logger and two FileHandlers. Add a Filter to each handler that only allows messages into the file that belong to that flow. You can use any information that exists on a LogRecord to decide where the message goes. The code below uses the extra kwarg of the logging methods but if there is something already inherent in the logs to decide the flow that is not required. Here are a few links to relevant parts of the documentation: FilterObjects LogRecord attributes

import logging

class Filter:
    def __init__(self, flow):
        self.flow = flow

    def filter(self, record):
        if record.flow == self.flow:
            return True

handlerA = logging.FileHandler('A.log')
handlerB = logging.FileHandler('B.log')
handlerA.addFilter(Filter('a'))
handlerB.addFilter(Filter('b'))

logger = logging.getLogger()
logger.addHandler(handlerA)
logger.addHandler(handlerB)

logger.warning('message A', extra={'flow':'a'}) # writes only to A.log
logger.warning('message B', extra={'flow':'b'}) # writes only to B.log

Using one logger with multiple file handlers that are almost same

1 Answers