Multiprocessing only works for the first iteration

问题

I am trying to use Python multiprocessing. I wrapped my statements inside a function then I used the multiprocessing map to loop over the function. I found that only the first iteration was really processed yet the rest did not ( I checked from that by printing the result).

Here are my problems :

1. Why only the first iteration was computed.
2. How to return each array separately B, C, and D.
3. My real calculations do have too many staff to calculate and to return, so is there is a more efficient than wrapping all my statement inside a function and then return them all. Thanks

import numpy as np
import multiprocessing as mp

B=np.full((5,4,4),np.nan)
C=np.full((5,4,4),np.nan)
D=np.full((5,4,4),np.nan)


def job1(i):
    print(i)
    A=np.ones((4,4))
    B[i,:,:]=A+1
    C[i,:,:]=2*A+2
    D[i,:,:]=A+5
    return B,C,D
#%%

P=mp.Pool(5)
result=P.map(job1,np.arange(5))
P.close()
P.join()



result[0] 
(array([[[ 2.,  2.,  2.,  2.],
         [ 2.,  2.,  2.,  2.],
         [ 2.,  2.,  2.,  2.],
         [ 2.,  2.,  2.,  2.]],

        [[nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan]],

        [[nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan]],

        [[nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan]],

        [[nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan],
         [nan, nan, nan, nan]]]),

回答1:

1. Your code works as expected. You have 5 processors (Pool(5)) and 5 things to do (np.arange(5)), so each task is executed by each processor. Each calculation is not shared due to the reason @Michael Butscher mentioned in the comment.

2. You can parse the result after you get it from Pool operation like below (an intuitive way);

output = {'B':[], 'C':[], 'D':[]}
for r in result:
    output['B'].append(r[0])
    output['C'].append(r[1])
    output['D'].append(r[2])

3. It is hard to figure out the most efficient way to process your jobs not seeing the reproducible codes. To run multiple functions, please refer the followed link;

Mulitprocess Pools with different functions

来源：https://stackoverflow.com/questions/57263630/multiprocessing-only-works-for-the-first-iteration