Context: I would like to create a single stream to stdout of log events formatted in JSON without any duplicate events. Within my own applications, this is relatively simple. I simply configure the root logger to have a single StreamHandler and use python-json-logger as the Formatter.
included_attributes = '%(asctime)s %(filename)s %(funcName)s %(lineno)d %(message)s %(levelname)s %(module)s %(name)s %(pathname)s %(process)d %(thread)s %(threadName)s'
json_formatter = jsonlogger.JsonFormatter(included_attributes)
def change_root_logger_format():
root_logger = logging.getLogger()
root_logger.setLevel(logging.DEBUG)
root_handler = logging.StreamHandler()
root_handler.setFormatter(json_formatter)
root_logger.handlers = [root_handler]
For various modules I add the standard logger = logging.getLogger(__name__).
All logs roll up the hierarchy to root with my single StreamHandler() and output to stdout as JSON.
As I understand it, best practice for distributed packages is that they don't declare their own handlers (or at least check if they are __main__ before doing so). I have found that this is not universally applied, however.
I am using at least one package that adds its own handlers. That is, the package __init__.py contains something like
logger = logging.getLogger(__name__)
logger.addHandler(StreamHandler())
As a result, I end up with duplicate logs for everything logged by this package. One line from its own handler, and one from the root logger.
The only solution I've found to this is to remove the handlers after the fact. That is, loop through logging.root.manager.loggerDict and removing the handlers of any Logger that isn't root. The problem with this approach is that any new import could cause new Handlers to appear. Thus, all of my modules would need the same boilerplate handler removal after their imports.
Does anyone know of a way to prevent this problem pre-emptively and exhuastively? That is, preventing all loggers but root from having Handlers?
I don't believe this question is a duplicate. There are many questions on disabling package/module loggers (for example, this one). These questions and answers don't apply to removing handlers while keeping the logger active.
