Share the list of lists in multiprocessing

问题

I want to increase the efficiency of my code. One intensive part of my code is to append elements to a list of lists. Basically, I want to do something as follows,

import multiprocessing
import time
def update_val(L, i):  
    L.append(i**2)
    return L

if __name__ == "__main__":
    N = 1000000
    x_reg = [list(range(10)) for i in range(N)]
    y_reg = [list(range(10)) for i in range(N)]
    z_reg = [list(range(10)) for i in range(N)]   
    "Regular Call"
    start = time.time()
    [x_reg[i].append(i**2)  for i in range(N)]
    stat_reg =time.time() - start

    "Multiprocessing"
    num_cores = multiprocessing.cpu_count() # equals 4 in my case
    pool = multiprocessing.Pool(num_cores)
    start = time.time()
    y_reg = pool.starmap(update_val,[(y_reg[i],i) for i in range(N)])
    pool.close()
    pool.join()
    stat_val =time.time() - start


    print("Regular: %g "%(stat_reg))
    print("Mult.: %g "%(stat_val))

The output is:

Regular: 0.387984 
Mult.: 2.54244 

I believe the reason is related to how multiprocessing works; It needs to make a copy of the original list, do staff and return it. This should be the main reason why multiprocessing is very slow in my case. Here is my question: 1- How should I perform the same function to z_reg while sharing it between processes? 2- Does it enhance the performance. 3- Any other idea how to attach something to every sub-list in z_reg?

回答1:

I suggest you use a multiprocessing.Queue. Keep the lists in the main process, and send the data from all children processes through the Queue.

来源：https://stackoverflow.com/questions/51272280/share-the-list-of-lists-in-multiprocessing