python multiprocessing/threading code exits early

问题

I'm trying to create multiple processes which each call multiple threads. I'm running the following code with python3.5

A simplified example of the problem looks like this:

import multiprocessing
import time
import threading    

class dumb(threading.Thread):
    def __init__(self):
        super(dumb, self).__init__()
    def run(self):
        while True:
            print("hi")
            time.sleep(1)    

def test():
    for i in range(2):
        bar = dumb()
        bar.start()
def main():
    p = []
    for i in range(2):
        p.append(multiprocessing.Process(target=test))
    for i in p:
        i.start()
    for i in p:
        i.join()
if __name__ == '__main__':
    main()

I expect this code to print "hi" forever, but it prints exactly once for each thread in each process (4 times total).

When I remove the multiprocessing, the multi-threading works.

When I remove the multi-threading, the multiprocessing works.

The part that I think is the problem after I read the multiprocessing documentation: The documentation for join states: "Block the calling thread until the process whose join() method is called terminates or until the optional timeout occurs."

If this was working as I expect it to, main would wait forever when trying to join.

I put try/except blocks around the while loop and didn't see any errors.

I have tried passing the "dumb" class a queue and passing exceptions through, but the queue stays empty.

Any tips on how to debug this would be greatly appreciated. My best guesses are

The thread is exiting early (print statements after while loop were never hit though)
Main exits and the processes are killed
Join is working properly, but not the way I expect it to?

Any ideas?

回答1:

Resolution: the new bug was closed as a duplicate of http://bugs.python.org/issue18966

Alas, there's no simple, satisfying explanation "for why". The cause is that multiprocessing arranges for worker processes to leave Python via calling os._exit() rather than the normal sys.exit(). os._exit() skips all "normal" shutdown processing. Part of what's skipped is .join()-ing non-daemon threads, so the process just vanishes while the threads are still running.

That should at least (according to me) be documented, or preferably changed.

In the meantime, the workaround - as you already know - is to explicitly .join() the threads yourself.

ANOTHER WAY

Under Python 3.4 or later, you could also use multiprocessing's spawn start method:

https://docs.python.org/3/library/multiprocessing.html?highlight=spawn#contexts-and-start-methods

That causes worker processes to finish via sys.exit(exitcode), which does all the normal shutdown processing (including .join()-ing non-daemon threads).

spawn is the only start method available on Windows, which explains why I had no problem running your original example.

来源：https://stackoverflow.com/questions/38356584/python-multiprocessing-threading-code-exits-early

标签

python

multithreading

python-3.x

python-multiprocessing