Python multiprocessing's Pool process limit

Question

In using the Pool object from the multiprocessing module, is the number of processes limited by the number of CPU cores? E.g. if I have 4 cores, even if I create a Pool with

Answer 1

Yes. Theoretically there is no limit on processes you can create, but an insane amount of processes started at once will cause death to the system because of the running out of memory. Note that processes occupy a much larger footprint than threads as they don't use shared space between them but use an individual space for each process.

so the best programming practice is to use semaphore restricted to the number of processors of your system. likely

pool = multiprocessing.Semaphore(4) # no of cpus of your system.

If you are not aware of the number of cores of your system or if you want to use the code in many systems, a generic code like the below will do...

pool = multiprocessing.Semaphore(multiprocessing.cpu_count()) 
#this will detect the number of cores in your system and creates a semaphore with that  value.  

P.S. But it is good to use number of cores-1 always.

Hope this helps :)