What is the most pythonic way of logging for multiple modules and multiple handlers with specified encoding?

Question

I\'m looking for concrete advice about how the multiple module and multiple handler logging should be done. I have added my simplified code here, but I don\'t want to bias t

Answer 1

For modules consisting of many parts, I use the method recommended in the documentation, which just has one line per module, logger = logging.getLogger(__name__). As you point out, the module shouldn't know or care how or where its messages are going, it just passes it on to the logger that should have been set up by the main program.

To reduce cut-n-paste depending on which is your main program, make sure your modules have a hierarchy that makes sense, and have just one function somewhere that sets up your logging, which can then be called by any main you wish.

For example, make a logsetup.py:

import logging

def configure_log(level=None,name=None):
    logger = logging.getLogger(name)
    logger.setLevel(level)

    file_handler = logging.FileHandler('../logs/%s' % name,'w','utf-8')
    file_handler.setLevel(logging.DEBUG)
    file_format = logging.Formatter('%(asctime)s %(levelname)s: %(message)s [in %(pathname)s:%(lineno)d in %(funcName)s]')
    file_handler.setFormatter(file_format)
    logger.addHandler(file_handler)

    console_handler = logging.StreamHandler()
    console_handler.setLevel(logging.INFO)
    console_format = logging.Formatter('%(message)s')
    console_handler.setFormatter(console_format)
    logger.addHandler(console_handler)

To have your individual modules have a mode in which they behave as main, define a separate function, such as main.

In level0.py and/or level1.py:

def main():
  # do whatever

And in the very top level program, call that function:

import logging
from logsetup import configure_log
configure_log(logging.DEBUG,'level0') # or 'level1'
from level0 import main # or level1

if __name__ == "__main__"
  main()

You should still have the __name__ == "__main__" clause there, some modules (cough multiprocessing cough) have different behavior depending on whether or not the clause exists.

Answer 2

To kinda wrap this up, here is what I did; I've put the following two lines in every module/file:

import logging
logger = logging.getLogger(__name__)

This sets up the logging, but doesn't add any handlers. I then add handlers to the root logger in the main file I'm running the imported modules therefore pass their records to the root logger and everything gets saved and displayed. I do it like CaptainMurphy suggested, but with logger = logging.getLogger('') to work with the root logger

To address encoding - I had problems saving non-ascii strings to file. So I just added FileHandler to the root logger, where encoding can be specified. I couldn't do it with logging.basicConfig.

Thank you again @CaptainMurphy - I'm sorry I can't upvote you with my low reputation, but I've checkmarked the answer.