Script stuck on exit when using atexit to terminate threads

问题

I'm playing around with threads on python 3.7.4, and I want to use atexit to register a cleanup function that will (cleanly) terminate the threads.

For example:

# example.py
import threading
import queue
import atexit
import sys

Terminate = object()

class Worker(threading.Thread):
    def __init__(self):
        super().__init__()
        self.queue = queue.Queue()

    def send_message(self, m):
        self.queue.put_nowait(m)

    def run(self):
        while True:
            m = self.queue.get()
            if m is Terminate:
                break
            else:
                print("Received message: ", m)


def shutdown_threads(threads):
    for t in threads:
        print(f"Terminating thread {t}")
        t.send_message(Terminate)
    for t in threads:
        print(f"Joining on thread {t}")
        t.join()
    else:
        print("All threads terminated")

if __name__ == "__main__":
    threads = [
        Worker()
        for _ in range(5)
    ]
    atexit.register(shutdown_threads, threads)

    for t in threads:
        t.start()

    for t in threads:
        t.send_message("Hello")
        #t.send_message(Terminate)

    sys.exit(0)

However, it seems interacting with the threads and queues in the atexit callback creates a deadlock with some internal shutdown routine:

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
^CException ignored in: <module 'threading' from '/usr/lib64/python3.7/threading.py'>
Traceback (most recent call last):
  File "/usr/lib64/python3.7/threading.py", line 1308, in _shutdown
    lock.acquire()
KeyboardInterrupt
Terminating thread <Worker(Thread-1, started 140612492904192)>
Terminating thread <Worker(Thread-2, started 140612484511488)>
Terminating thread <Worker(Thread-3, started 140612476118784)>
Terminating thread <Worker(Thread-4, started 140612263212800)>
Terminating thread <Worker(Thread-5, started 140612254820096)>
Joining on thread <Worker(Thread-1, stopped 140612492904192)>
Joining on thread <Worker(Thread-2, stopped 140612484511488)>
Joining on thread <Worker(Thread-3, stopped 140612476118784)>
Joining on thread <Worker(Thread-4, stopped 140612263212800)>
Joining on thread <Worker(Thread-5, stopped 140612254820096)>
All threads terminated

(the KeyboardInterrupt is me using ctrl-c since the process seems to be hanging indefinitely).

However, if I send the Terminate message before exit(uncomment the line after t.send_message("Hello")), the program doesn't hang and terminates gracefully:

$ python example.py
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Received message:  Hello
Terminating thread <Worker(Thread-1, stopped 140516051592960)>
Terminating thread <Worker(Thread-2, stopped 140516043200256)>
Terminating thread <Worker(Thread-3, stopped 140515961992960)>
Terminating thread <Worker(Thread-4, stopped 140515953600256)>
Terminating thread <Worker(Thread-5, stopped 140515945207552)>
Joining on thread <Worker(Thread-1, stopped 140516051592960)>
Joining on thread <Worker(Thread-2, stopped 140516043200256)>
Joining on thread <Worker(Thread-3, stopped 140515961992960)>
Joining on thread <Worker(Thread-4, stopped 140515953600256)>
Joining on thread <Worker(Thread-5, stopped 140515945207552)>
All threads terminated

This begs the question, when does this threading._shutdown routine gets executed, relative to atexit handlers? Does it make sense to interact with threads in atexit handlers?

回答1:

atexit.register(func) registers func as a function to be executed at termination.

After execute the last line of code (it is sys.exit(0) in above example) in main thread, threading._shutdown was invoked (by interpreter) to wait for all non-daemon threads (Workers created in above example) exit