Using python multiprocessing Pool in the terminal and in code modules for Django or Flask

Question

When using multiprocessing.Pool in python with the following code, there is some bizarre behavior.

from multiprocessing import Pool
p = Pool(3)
def f(x): ret         


        

Answer 1

The function you want to execute on a thread pool must already be defined when you create the pool.

This should work:

from multiprocessing import Pool
def f(x): print(x)
if __name__ == '__main__':
    p = Pool(3)
    p.map(f, range(20))

The reason is that (at least on systems having fork) when you create a pool the workers are created by forking the current process. So if the target function isn't already defined at that point, the worker won't be able to call it.

On windows it's a bit different, as windows doesn't have fork. Here new worker processes are started and the main module is imported. That's why on windows it's important to protect the executing code with a if __name__ == '__main__'. Otherwise each new worker will reexecute the code and therefore spawn new processes infinitely, crashing the program (or the system).