Python Logging - Disable logging from imported modules

Question

I\'m using the Python logging module, and would like to disable log messages printed by the third party modules that I import. For example, I\'m using something like the fo

Answer 1

I had the same problem. I have a logging_config.py file which I import in all other py files. In logging_config.py file I set root logger logging level to ERROR (by default its warning):

logging.basicConfig(
    handlers=[
        RotatingFileHandler('logs.log',maxBytes=1000, backupCount=2),
        logging.StreamHandler(), #print to console
    ],
    level=logging.ERROR
)

In other modules I import logging_config.py and declare a new logger and set its level to debug:

log = logging.getLogger(__name__)
log.setLevel(logging.DEBUG)

This way everything I log in my py files is logged, but stuff logged at debug and info level by imported modules like urllib, request,boto3 etc is not logged. If there is some error in those import module then its logged, since I set root loggers level to ERROR.

Answer 2

If you're going to use the python logging package, it's a common convention to define a logger in every module that uses it.

logger = logging.getLogger(__name__)

Many popular python packages do this, including requests. If a package uses this convention, it's easy to enable/disable logging for it, because the logger name will be the same name as the package (or will be a child of that logger). You can even log it to the same file as your other loggers.

logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)

requests_logger = logging.getLogger('requests')
requests_logger.setLevel(logging.DEBUG)

handler = logging.StreamHandler()
handler.setLevel(logging.DEBUG)
logger.addHandler(handler)
requests_logger.addHandler(handler)

Answer 3

Simply doing something like this solves the problem:

logging.config.dictConfig({'disable_existing_loggers': True,})

NOTE: Wherever you're placing this line, make sure all the imports everywhere in your project are done within that file itself. :)

Answer 4

Another thing to consider is the propagate property of the Logger class.

For example, py-suds library for handling soap calls, even put to ERROR

logging.getLogger('suds.client').setLevel(logging.ERROR)
logging.getLogger('suds.transport').setLevel(logging.ERROR)
logging.getLogger('suds.xsdschema').setLevel(logging.ERROR)
logging.getLogger('suds.wsdl').setLevel(logging.ERROR)

logs logs about a module called sxbasics.py creationg a huge amount of logs

that because the propagation of the logs is True by default, setting to False, instead, i recovered 514MB of logs.

import logging
logging.getLogger("suds").propagate = False
logging.getLogger('suds.client').setLevel(logging.ERROR)
logging.getLogger('suds.transport').setLevel(logging.ERROR)
logging.getLogger('suds.xsdschema').setLevel(logging.ERROR)
logging.getLogger('suds.wsdl').setLevel(logging.ERROR)

Answer 5

The problem is that calling getLogger without arguments returns the root logger so when you set the level to logging.DEBUG you are also setting the level for other modules that use that logger.

You can solve this by simply not using the root logger. To do this just pass a name as argument, for example the name of your module:

logger = logging.getLogger('my_module_name')
# as before

this will create a new logger and thus it wont inadvertently change logging level for other modules.

---

Obviously you have to use logger.debug instead of logging.debug since the latter is a convenience function that calls the debug method of the root logger.

This is mentioned in the Advanced Logging Tutorial. It also allows you to know which module triggered the log message in a simple way.

Answer 6

@Bakuriu quite elegantly explains the function. Conversely, you can use the getLogger() method to retrieve and reconfigure/disable the unwanted loggers.

I also wanted to add the logging.fileConfig() method accepts a parameter called disable_existing_loggers which will disable any loggers previously defined (i.e., in imported modules).