Multiprocessing in Python while limiting the number of running processes

Question

I\'d like to run multiple instances of program.py simultaneously, while limiting the number of instances running at the same time (e.g. to the number of CPU cores available on m

Answer 1

Bash script rather than Python, but I use it often for simple parallel processing:

#!/usr/bin/env bash
waitForNProcs()
{
 nprocs=$(pgrep -f $procName | wc -l)
 while [ $nprocs -gt $MAXPROCS ]; do
  sleep $SLEEPTIME
  nprocs=$(pgrep -f $procName | wc -l)
 done
}
SLEEPTIME=3
MAXPROCS=10
procName=myPython.py
for file in ./data/*.txt; do
 waitForNProcs
 ./$procName $file &
done

Or for very simple cases, another option is xargs where P sets the number of procs

find ./data/ | grep txt | xargs -P10 -I SUB ./myPython.py SUB