Python Logging - Disable logging from imported modules

Question

I'm using the Python logging module, and would like to disable log messages printed by the third party modules that I import. For example, I'm using something like the following:

logger = logging.getLogger()
logger.setLevel(level=logging.DEBUG)
fh = logging.StreamHandler()
fh_formatter = logging.Formatter('%(asctime)s %(levelname)s %(lineno)d:%(filename)s(%(process)d) - %(message)s')
fh.setFormatter(fh_formatter)
logger.addHandler(fh)

This prints out my debug messages when I do a logger.debug("my message!"), but it also prints out the debug messages from any module I import (such as requests, and a number of other things).

I'd like to see only the log messages from modules I'm interested in. Is it possible to make the logging module do this?

Ideally, I'd like to be able tell the logger to print messages from "ModuleX, ModuleY" and ignore all others.

I looked at the following, but I don't want to have to disable/enable logging before every call to an imported function: logging - how to ignore imported module logs?

Answer 1

The problem is that calling getLogger without arguments returns the root logger so when you set the level to logging.DEBUG you are also setting the level for other modules that use that logger.

You can solve this by simply not using the root logger. To do this just pass a name as argument, for example the name of your module:

logger = logging.getLogger('my_module_name')
# as before

this will create a new logger and thus it wont inadvertently change logging level for other modules.

---

Obviously you have to use logger.debug instead of logging.debug since the latter is a convenience function that calls the debug method of the root logger.

This is mentioned in the Advanced Logging Tutorial. It also allows you to know which module triggered the log message in a simple way.

Answer 2

If you're going to use the python logging package, it's a common convention to define a logger in every module that uses it.

logger = logging.getLogger(__name__)

Many popular python packages do this, including requests. If a package uses this convention, it's easy to enable/disable logging for it, because the logger name will be the same name as the package (or will be a child of that logger). You can even log it to the same file as your other loggers.

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

requests_logger = logging.getLogger('requests')
requests_logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
requests_logger.addHandler(handler)

Answer 3

Not sure if this is appropriate to post, but I was stuck for a long time & wanted to help out anyone with the same issue, as I hadn't found it anywhere else!

I was getting debug logs from matplotlib despite following the pretty straightforward documentation at the logging advanced tutorial and the troubleshooting. I was initiating my logger in main() of one file and importing a function to create a plot from another file (where I had imported matplotlib).

What worked for me was setting the level of matplotlib before importing it, rather than after as I had for other modules in my main file. This seemed counterintuitive to me so if anyone has insight into how you can set the config for a logger that hasn't been imported yet I'd be curious to find out how this works. Thanks!

In my main file:

import logging
import requests
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logging.getLogger('requests').setLevel(logging.DEBUG)

def main():
  ...

In my plot.py file:

import logging
logging.getLogger('matplotlib').setLevel(logging.WARNING)
import matplotlib.pyplot as plt

def generatePlot():
  ...

Answer 4

This disables all existing loggers, such as those created by imported modules, while still using the root logger (and without having to load an external file).

import logging.config
logging.config.dictConfig({
    'version': 1,
    'disable_existing_loggers': True,
})

Note that you need to import all modules you don't want logged first! Otherwise those won't be considered as "existing loggers". It will then disable all loggers from those modules. This might lead you to also miss out on important errors!

For more detailed examples using related options for configuration, see https://gist.github.com/st4lk/6287746, and here is a (partially working) example using YAML for config with the coloredlog library.

Answer 5

@Bakuriu quite elegantly explains the function. Conversely, you can use the getLogger() method to retrieve and reconfigure/disable the unwanted loggers.

I also wanted to add the logging.fileConfig() method accepts a parameter called disable_existing_loggers which will disable any loggers previously defined (i.e., in imported modules).

Answer 6

You could use something like:

logging.getLogger("imported_module").setLevel(logging.WARNING)
logging.getLogger("my_own_logger_name").setLevel(logging.DEBUG)

This will set my own module's log level to DEBUG, while preventing the imported module from using the same level.

Note: "imported_module" can be replaced with imported_module.__name__ (without quotes), and "my_own_logger_name" can be replaced by __name__ if that's the way you prefer to do it.

Answer 7

Simply doing something like this solves the problem:

logging.config.dictConfig({'disable_existing_loggers': True,})

NOTE: Wherever you're placing this line, make sure all the imports everywhere in your project are done within that file itself. :)

Answer 8

I had the same problem. I have a logging_config.py file which I import in all other py files. In logging_config.py file I set root logger logging level to ERROR (by default its warning):

logging.basicConfig(
    handlers=[
        RotatingFileHandler('logs.log',maxBytes=1000, backupCount=2),
        logging.StreamHandler(), #print to console
    ],
    level=logging.ERROR
)

In other modules I import logging_config.py and declare a new logger and set its level to debug:

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

This way everything I log in my py files is logged, but stuff logged at debug and info level by imported modules like urllib, request,boto3 etc is not logged. If there is some error in those import module then its logged, since I set root loggers level to ERROR.

Answer 9

Another thing to consider is the propagate property of the Logger class.

For example, py-suds library for handling soap calls, even put to ERROR

logging.getLogger('suds.client').setLevel(logging.ERROR)
logging.getLogger('suds.transport').setLevel(logging.ERROR)
logging.getLogger('suds.xsdschema').setLevel(logging.ERROR)
logging.getLogger('suds.wsdl').setLevel(logging.ERROR)

logs logs about a module called sxbasics.py creationg a huge amount of logs

that because the propagation of the logs is True by default, setting to False, instead, i recovered 514MB of logs.

import logging
logging.getLogger("suds").propagate = False
logging.getLogger('suds.client').setLevel(logging.ERROR)
logging.getLogger('suds.transport').setLevel(logging.ERROR)
logging.getLogger('suds.xsdschema').setLevel(logging.ERROR)
logging.getLogger('suds.wsdl').setLevel(logging.ERROR)

Answer 10

After trying various answers in this thread and other forums, I found this method efficient at silencing other modules' loggers. This was inspired by the following link:

https://kmasif.com/2019-11-12-ignore-logging-from-imported-module/

import logging
# Initialize your own logger
logger = logging.getLogger('<module name>')
logger.setLevel(logging.DEBUG)

# Silence other loggers
for log_name, log_obj in logging.Logger.manager.loggerDict.items():
     if log_name != '<module name>':
          log_obj.disabled = True

Note that you would want to do this after importing other modules. However, I found this to be a convenient and fast way to disabled other modules' loggers.