How to log messages from different threads to different files?

问题

I have a Driver.py scripts where it calls multiple threads based on the given inputs. Threads are basically runs a module of a selected object. So Driver.py may call thread_1.run(), thread_2.run(), thread_3.run(), and continue its process.

Driver.py logs its output into a main.log folder, where I want threads to log their output into unique filenames for each. Driver.py and Threads also uses common modules that are defined on differnt files, where they also logs information.

I call setup_load("main.log") first on Driver, afterwards in each Thread, called setup_load(f"thread_{jobid}.log") as well. I realize that when Thread is called, now Driver.py writes into thread's log file. I may use a differnt logger inside Thread but when the Thread calls another modules, since those common modules are using import logging they write into the root logger's defined filename.

=> Is it possible to log messages from different threads to different files? I found multiple answers on SO (for example), but non of them covers when another module is called on a different file, how would they can find out that which logger they can use.

=> So the problem I am facing is since every thread is using the same underlying logger, when I change the file path of the logging.basicConfig in one thread, it affects the class across all threads and the driver, since they're all using it.

=> How would be functions from different modules called from the thread or driver would understand which logger should it choose?

Comment section on How to change filehandle with Python logging on the fly with different classes and imports has a discussion and recommended solution.

@Martijn Pieters:

next option: create per-thread handlers, give each of them a filter that filters on the logrecord thread attribute. Attach a filter to any other handlers that returns False for logrecords with thread set

回答1:

Yes, you can direct log entries from different threads to different files. You'll need to:

Create a log filter that can filter records by their LogRecord.thread or LogRecord.threadName attribute
Create a filter that does not accept records with specific or all thread ids.
Create a log handler per thread, giving it a log filter that only accepts logrecords for their specific thread.
Attach the filter that ignores log records for your threads to any other handlers.

When filtering, you have the choice between filtering on thread id (the value returned by threading.get_ident()) or thread name (whatever you passed in as the name argument to the Thread() object). If you have a pattern for your thread names, this is where you'd use it.

Creating a custom filter is easy enough:

import threading
from logging import Filter

class ThreadFilter(Filter):
    """Only accept log records from a specific thread or thread name"""

    def __init__(self, threadid=None, threadname=None):
        if threadid is None and threadname is None:
            raise ValueError("Must set at a threadid and/or threadname to filter on")
        self._threadid = threadid
        self._threadname = threadname

    def filter(self, record):
        if self._threadid is not None and record.thread != self._threadid:
            return False
        if self._threadname is not None and record.threadName != self._threadname:
            return False
        return True

class IgnoreThreadsFilter(Filter):
    """Only accepts log records that originated from the main thread"""

    def __init__(self):
        self._main_thread_id = threading.main_thread().ident

    def filter(self, record):
        return record.thread == self._main_thread_id

If you want to match a specific pattern, adjust the code accordingly.

来源：https://stackoverflow.com/questions/61818674/how-to-log-messages-from-different-threads-to-different-files

标签

python

logging