Python 3 Multiprocessing queue deadlock when calling join before the queue is empty

Question

I have a question understanding the queue in the multiprocessing module in python 3

This is what they say in the programming guidelines:

Answer 1

Don't call join() on a process object before you got all messages from the shared queue.

I used following workaround to allow processes to exit before processing all its results:

results = []
while True:
    try:
        result = resultQueue.get(False, 0.01)
        results.append(result)
    except queue.Empty:
        pass
    allExited = True
    for t in processes:
        if t.exitcode is None:
            allExited = False
            break
    if allExited & resultQueue.empty():
        break

It can be shortened but I left it longer to be more clear for newbies.

Here resultQueue is the multiprocess.Queue that was shared with multiprocess.Process objects. After this block of code you will get the result array with all the messages from the queue.

The problem is that input buffer of the queue pipe that receive messages may become full causing writer(s) infinite block until there will be enough space to receive next message. So you have three ways to avoid blocking:

• Increase the multiprocessing.connection.BUFFER size (not so good)
• Decrease message size or its amount (not so good)
• Fetch messages from the queue immediately as they come (good way)