Forking / Multi-Threaded Processes | Bash

Question

I would like to make a section of my code more efficient. I\'m thinking of making it fork off into multiple processes and have them execute 50/100 times at once, instead of

Answer 1

I don't like using wait because it gets blocked until the process exits, which is not ideal when there are multiple process to wait on as I can't get a status update until the current process is done. I prefer to use a combination of kill -0 and sleep to this.

Given an array of pids to wait on, I use the below waitPids() function to get a continuous feedback on what pids are still pending to finish.

declare -a pids
waitPids() {
    while [ ${#pids[@]} -ne 0 ]; do
        echo "Waiting for pids: ${pids[@]}"
        local range=$(eval echo {0..$((${#pids[@]}-1))})
        local i
        for i in $range; do
            if ! kill -0 ${pids[$i]} 2> /dev/null; then
                echo "Done -- ${pids[$i]}"
                unset pids[$i]
            fi
        done
        pids=("${pids[@]}") # Expunge nulls created by unset.
        sleep 1
    done
    echo "Done!"
}

When I start a process in the background, I add its pid immediately to the pids array by using this below utility function:

addPid() {
    local desc=$1
    local pid=$2
    echo "$desc -- $pid"
    pids=(${pids[@]} $pid)
}

Here is a sample that shows how to use:

for i in {2..5}; do
    sleep $i &
    addPid "Sleep for $i" $!
done
waitPids

And here is how the feedback looks:

Sleep for 2 -- 36271
Sleep for 3 -- 36272
Sleep for 4 -- 36273
Sleep for 5 -- 36274
Waiting for pids: 36271 36272 36273 36274
Waiting for pids: 36271 36272 36273 36274
Waiting for pids: 36271 36272 36273 36274
Done -- 36271
Waiting for pids: 36272 36273 36274
Done -- 36272
Waiting for pids: 36273 36274
Done -- 36273
Waiting for pids: 36274
Done -- 36274
Done!