How can I restrict the scope of a multiprocessing process?

Question

Using python\'s multiprocessing module, the following contrived example runs with minimal memory requirements:

import multiprocessing 
# completely_unrelated         


        

Answer 1

What is important here is which platform you are targeting. Unix systems processes are created by using Copy-On-Write (cow) memory. So even though each process gets a copy of the full memory of the parent process, that memory is only actually allocated on a per page bases (4KiB)when it is modified. So if you are only targeting these platforms you don't have to change anything.

If you are targeting platforms without cow forks you may want to use python 3.4 and its new forking contexts spawn and forkserver, see the documentation These methods will create new processes which share nothing or limited state with the parent and all memory passing is explicit.

But not that that the spawned process will import your module so all global data will be explicitly copied and no copy-on-write is possible. To prevent this you have to reduce the scope of the data.

import multiprocessing  as mp
import numpy as np

def foo(x):
    import time
    time.sleep(60)

if __name__ == "__main__":
    mp.set_start_method('spawn')
    # not global so forks will not have this allocated due to the spawn method
    # if the method would be fork the children would still have this memory allocated
    # but it could be copy-on-write
    completely_unrelated_array = np.ones((5000, 10000))
    P = mp.Pool()
    for x in range(3):
        mp.Process(target=foo, args=(x,)).start()

e.g top output with spawn:

%MEM     TIME+ COMMAND
29.2   0:00.52 python3                                                
0.5   0:00.00 python3    
0.5   0:00.00 python3    
0.5   0:00.00 python3    

and with fork:

%MEM     TIME+ COMMAND
29.2   0:00.52 python3                                                
29.1   0:00.00 python3    
29.1   0:00.00 python3                                                
29.1   0:00.00 python3

note how its more than 100%, due to copy-on-write