Should I use Events, Semaphores, Locks, Conditions, or a combination thereof to manage safely exiting my multithreaded Python program?

问题

I am writing a multithreaded python program in which the main thread and the other threads it spawns run as daemons (but not with Thread.daemon=True) that look for certain files in certain directories and perform actions with them when they exist. It is possible that an error occurs in one/any of the threads which would require the whole program to exit. However, I need the other threads to finish their current job before exiting.

From what I understand, if I set myThread.daemon=True for my spawned threads, they will automatically exit immediately when the main thread exits. However, I want the other threads to finish their current job before exiting (unless the error is some sort of catastrophic failure, in which case I'll probably just exit everything anyway, safely or not). Therefore, I am not setting the daemon property to True for the threads.

Looking at the threading module documentation and the various objects available to me such as Events, Semaphores, Conditions, and Locks, I'm unsure of the best way to handle my situation. Additionally, I'm unsure how to handle this scenario when the program needs to terminate due to SIGTERM/SIGINT signals.

Some code that illustrates a simplified version of the structure of my program:

import threading
import signals
import glob
import time

class MyThread1( threading.thread ):
    def __init__( self, name='MyThread1' ):
        threading.Thread.__init__( self )
        self.name = name
        return
    def run( self ):
        while True:
            filePathList = glob.glob( thisThreadDir + '/*.txt' )
            for file in filePathList:
                try:
                    doSomeProcessing( file )
                    # Then move the file to another thread's dir
                    # or potentially create a new file that will 
                    # be picked up by another thread
                except:
                    # Need to potentially tell all other threads
                    # to finish task and exit depending on error

            # I assume this would be the place to check for some kind of
            # flag or other indication to terminate the thread?
            time.sleep( 30 )


# Now imagine a few more custom threads with the same basic structure, 
# except that what is happening in doSomeProcessing() will obviously vary

# Main Thread/Script
def sigintHandler( SIGINT, frame ):
    # How do I handle telling all threads to finish their current loop
    # and then exit safely when I encounter this signal?
    sys.exit( 1 )

def sigtermHandler( SIGTERM, frame ):
    # Same question for this signal handler
    sys.exit( 1 )

signal.signal( signal.SIGINT, sigintHandler )
signal.signal( signal.SIGTERM, sigtermHandler )

myOtherThread1 = MyThread1()
myOtherThreadN = MyThreadN()

myOtherThread1.start()
myOtherThreadN.start()

while True:
    filePathList = glob.glob( mainDir + '/*.txt' )
    for file in filePathList:
        try:
            doMainProcessing( file )
            # Move file or write a new one in another thread's dir
        except:
            # Again, potentially need to exit the whole program, but want 
            # the other threads to finish their current loop first 

    # Check if another thread told us we need to exit?
    time.sleep( 30 )

回答1:

I would use an Event to signal to a thread that it should exit:

create an event in __init__
use the event's wait() in run() for sleep and for checking when to exit
set the event from outside to stop the thread

To handle exceptions within a thread, I would have a try/ except block around everything it does. When something is caught, store the exception (and/or any other info you need), clean up and exit the thread.

Outside, in the main thread, check for the store exceptions in all threads, if any exception is found, signal to all threads that they should exit.

To handle exceptions in the main thread (which includes also SIGINT), have a try/except block there and signal to all threads to stop.

All together, with dummy exceptions and debug prints:

import threading
import time

class MyThread(threading.Thread):
    def __init__(self):
        super().__init__()
        self.stop_requested = threading.Event()
        self.exception = None

    def run(self):
        try:
            # sleep for 1 second, or until stop is requested, stop if stop is requested
            while not self.stop_requested.wait(1):
                # do your thread thing here
                print('in thread {}'.format(self))

                # simulate a random exception:
                import random
                if random.randint(0, 50) == 42:
                    1 / 0
        except Exception as e:
            self.exception = e

        # clean up here
        print('clean up thread {}'.format(self))

    def stop(self):
        # set the event to signal stop
        self.stop_requested.set()

# create and start some threads
threads = [MyThread(), MyThread(), MyThread(), MyThread()]
for t in threads:
    t.start()

# main thread looks at the status of all threads
try:
    while True:
        for t in threads:
            if t.exception:
                # there was an error in a thread - raise it in main thread too
                # this will stop the loop
                raise t.exception
        time.sleep(0.2)

except Exception as e:
    # handle exceptions any way you like, or don't
    # This includes exceptions in main thread as well as those in other threads
    # (because of "raise t.exception" above)
    print(e)

finally:
    print('clan up everything')
    for t in threads:
        # threads will know how to clean up when stopped
        t.stop()

来源：https://stackoverflow.com/questions/50731203/should-i-use-events-semaphores-locks-conditions-or-a-combination-thereof-to

标签

python

multithreading

python-3.x

python-multithreading