Python how to kill threads blocked on queue with signals?

问题

I start a bunch of threads working on a queue and I want to kill them when sending the SIGINT (Ctrl+C). What is the best way to handle this?

targets = Queue.Queue()
threads_num = 10
threads = []

for i in threads_num:
    t = MyThread()
    t.setDaemon(True)
    threads.append(t)
    t.start()

targets.join()

回答1:

Isn't Ctrl+C SIGINT?

Anyway, you can install a handler for the appropriate signal, and in the handler:

set a global flag that instructs the workers to exit, and make sure they check it periodically
or put 10 shutdown tokens on the queue, and have the workers exit when they pop this magic token
or set a flag which instructs the main thread to push those tokens, make sure the main thread checks that flag

etc. Mostly it depends on the structure of the application you're interrupting.

回答2:

If you are not interested in letting the other threads shut down gracefully, simply start them in daemon mode and wrap the join of the queue in a terminator thread.

That way, you can make use of the join method of the thread -- which supports a timeout and does not block off exceptions -- instead of having to wait on the queue's join method.

In other words, do something like this:

term = Thread(target=someQueueVar.join)
term.daemon = True
term.start()
while (term.isAlive()):
    term.join(3600)

Now, Ctrl+C will terminate the MainThread whereupon the Python Interpreter hard-kills all threads marked as "daemons". Do note that this means that you have to set "Thread.daemon" for all the other threads or shut them down gracefully by catching the correct exception (KeyboardInterrupt or SystemExit) and doing whatever needs to be done for them to quit.

Do also note that you absolutely need to pass a number to term.join(), as otherwise it will, too, ignore all exceptions. You can select an arbitrarily high number, though.

回答3:

One way to do it is to install a signal handler for SIGTERM that directly calls os._exit(signal.SIGTERM). However unless you specify the optional timeout argument to Queue.get the signal handler function will not run until after the get method returns. (That's completely undocumented; I discovered that on my own.) So you can specify sys.maxint as the timeout and put your Queue.get call in a retry loop for purity to get around that.

回答4:

Why don't you set timeouts for any operation on the queue? Then your threads can regular check if they have to finish by checking if an Event is raised.

回答5:

I managed to solve the problem by emptying the queue on KeyboardInterrupt and letting threads to gracefully stop themselves.

I don't know if it's the best way to handle this but is simple and quite clean.

targets = Queue.Queue()
threads_num = 10
threads = []

for i in threads_num:
    t = MyThread()
    t.setDaemon(True)
    threads.append(t)
    t.start()

while True:
    try:
        # If the queue is empty exit loop
        if self.targets.empty() is True:
            break

    # KeyboardInterrupt handler
    except KeyboardInterrupt:
        print "[X] Interrupt! Killing threads..."
        # Substitute the old queue with a new empty one and exit loop
        targets = Queue.Queue()
        break

# Join every thread on the queue normally
targets.join()

来源：https://stackoverflow.com/questions/7610545/python-how-to-kill-threads-blocked-on-queue-with-signals

标签

python

multithreading

queue