multiprocessing freeze computer

问题

I improved my execution time by using multiprocessing but I am not sure whether the behavior of the PC is correct, it freezes the system until all processes are done. I am using Windows 7 and Python 2.7.

Perhaps I am doing a mistake, here is what I did:

def do_big_calculation(sub_list, b, c):
    # do some calculations here with the sub_list

if __name__ == '__main__':
    list = [[1,2,3,4], [5,6,7,8], [9,10,11,12]]
    jobs = []
    for sub_l in list :
        j = multiprocessing.Process(target=do_big_calculation, args=(sub_l, b, c))
        jobs.append(j)
    for j in jobs:
        j.start()

回答1:

Here, you are creating 1 Process per task. This is will run all your tasks in parallel but it imposes an heavy overhead on your computer as your scheduler will need to manage many processes. This can cause a system freeze as too many ressources are used for your program.

A solution here could be to use the multiprocessing.Pool to run a given number of processes simultaneously performing some tasks:

import multiprocessing as mp

def do_big_calculation(args):
    sub_list, b, c = args
    return 1

if __name__ == '__main__':
    b, c = 1, 1
    ll = [([1, 2, 3, 4], b, c),
          ([5, 6, 7, 8], b, c),
          ([9, 10, 11, 12], b, c)]
    pool = mp.Pool(4)
    result = pool.map(do_big_calculation, ll)
    pool.terminate()
    print(result)

If you are ready to use third party library, you could also take a look at concurrent.futures (you need to install it in python2.7 but it exists for python3.4+) or joblib (available with pip):

from joblib import Parallel, delayed

def do_big_calculation(sub_list, b, c):
    return 1

if __name__ == '__main__':
    b, c = 1, 1
    ll = [([1, 2, 3, 4], b, c),
          ([5, 6, 7, 8], b, c),
          ([9, 10, 11, 12], b, c)]
    result = Parallel(n_jobs=-1)(
        delayed(do_big_calculation)(l, b, c) for l in ll)
    print(result)

The main advantage of such library is that it is developing whereas multiprocessing in python2.7 is freezed. Thus, there are bug fixes and improvements relatively often.
It also implements some clever tools to reduce the overhead for the computation. For instance, it uses memory mapping for big numpy array (reducing the memory footprint of starting all jobs).

来源：https://stackoverflow.com/questions/39892551/multiprocessing-freeze-computer

标签

python

windows

python-multiprocessing