Shell, run four processes parallel

Question

Currently I\'m stuck at running time consuming simulations efficiently. The intention is to run 4 simulations in parallel, because it\'s a single thread application and a qu

Answer 1

NUMJOBS=30
NUMPOOLS=4

seq 1 "$NUMJOBS" | for p in $(seq 1 $NUMPOOLS); do
    while read x; do ./sim -r "$x"; done &
done

The for loop creates a pool of background processes which reads from the shared standard input to start a simulation. Each background process "blocks" while its simulation is running, then reads the next job number from the seq command.

Without the for loop, it might be a little easier to follow:

seq 1 "$NUMJOBS" | {
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
    while read x; do ./sim -r "$x"; done &
}

Assuming sim takes a non-trivial amount of time to run, the first while will read 1 from its standard input, the 2nd 2, etc. Whichever sim finishes first, that while loop will read 5 from standard input; the next to finish will read 6, and so on. Once the last simulation is started, each read will fail, causing the loop to exit.