Restarting Upstart instance processes

Question

I am running multiple instances of a worker as described in this answer: Starting multiple upstart instances automatically

Question: Can I restart all instan

Answer 1

Essentially you need to have a process that executes many stop and start commands for all your N=1, N=2 combination.

A simple way to do this is a couple of bash for loops inside an exec script stanza. However, if the processes take some time to stop (e.g. because they are working on something and they are accepting SIGTERM after having processed their current job) this is inefficient as you have to wait for one to stop before sending the signal to the next one.

Therefore, I built an Upstart script that stops them in parallel at https://github.com/elifesciences/builder-base-formula/blob/master/elife/config/etc-init-multiple-processes-parallel.conf

The script is compiled by Salt using as input a map of process names to how many are there. Here is a sample result:

description "(Re)starts all instances, in parallel"
# http://upstart.ubuntu.com/cookbook/#start-on
start on (local-filesystems and net-device-up IFACE!=lo)
task
script
    timeout=300
    echo "--------"

    echo "Current status of 5 elife-bot-worker processes"
    echo "Now is" $(date -Iseconds)
    for i in `seq 1 5`
    do
        status elife-bot-worker ID=$i || true
    done
    echo "Stopping asynchronously 5 elife-bot-worker processes"
    echo "Now is" $(date -Iseconds)
    for i in `seq 1 5`
    do
        (stop elife-bot-worker ID=$i &) || true
    done

    for i in `seq 1 5`
    do
        echo "Waiting for elife-bot-worker $i to stop"
        echo "Now is" $(date -Iseconds)
        counter=0
        while true
        do
            if [ "$counter" -gt "$timeout" ]
            then
                echo "It shouldn't take more than $timeout seconds to kill all the elife-bot-worker processes"
                exit 1
            fi
            status elife-bot-worker ID=$i 2>&1 | grep "Unknown instance" && break
            sleep 1
            counter=$((counter + 1))
        done
    done
    echo "Stopped all elife-bot-worker processes"

    echo "Starting 5 elife-bot-worker processes"
    for i in `seq 1 5`
    do
        start elife-bot-worker ID=$i
    done
    echo "Started 5 elife-bot-worker processes"

end script