Using a loop variable in a Bash script to pass different command-line arguments

Question

I have a C++ program to which I pass two doubles as inputs from the command line using

int main(int argc, char *argv[]){
    double a,b;
    a = atof(argv[1]);
    b = atof(argv[2]);
    further code.....

I run the code on a cluster using the qsub utility and I have a Bash script named 'jobsub.sh` to submit the jobs which looks like this:

#!/bin/csh -f
hostname
cd /home/roy/codes/3D             # Change directory first -- replace Mysubdir
set startdir = `pwd`               # Remember the directory we're in
if( ! -d /scratch/$USER ) then
    mkdir /scratch/$USER       # Create scratch directory
endif                              # If it does not exist
#cp infile12 /scratch/$USER     # Copy input file to scratch directory
cd /scratch/$USER                  # Change to scratch directory
#rm *.*
$HOME/codes/3D/autoA100.out 2.1 2.2          # Run a program
cp * $startdir         # Copy outputfiles back to where we started

At the terminal I do qsub jobsub.sh.

However, I want to run the same executable for different values of a and b in parallel on different cores. Is it possible to write a for loop in the Bash script so that I can do something like,

for i=1;i<=10;i++ {
   $HOME/codes/3D/autoA100.out 2+i*0.1 2+i*0.2
}

Answer 1

If you are submitting the execution script to a batch loader, then there is no way to have a simple loop do the execution like you want because the entire script is run on each node. However, most batch loaders provide environment variables to the user.

PBS, for example, has $PBS_ARRAYID, which specifies the unique ID for the job as it is running. So instead of using a loop, your script can have:

a=$(echo "2+($PBS_ARRAYID+1)*0.1" | bc -l)
b=$(echo "2+($PBS_ARRAYID+1)*0.2" | bc -l)
$HOME/codes/3D/autoA100.out $a $b

Notice I've added 1 to $PBS_ARRAYID above because the ID begins from 0 while your loop begins from 1. It's also worth mentioning that while bash can do some arithmetic natively, it cannot handle real numbers; that's why I had to invoke bc.

Answer 2

You can always run jobs in the background (in which case the shell will continue with the next instruction)

MY_PIDS=
for i in 1 2 3 4 5 6 7 8 9 10
do
    yourjob "$(a i)" "$(b i)" &   # Note the ampersand (background)
    MY_PIDS="$MY_PIDS "$!         # Do not write the $! inside the quotes, bash will
                                  # break (better, don't use bash but plain sh)
done

# Wait wor each job to complete and cleanup
for p in $MY_PIDS
do
    wait "$p"
    echo "Return code of job $p is $?"
    do_cleanup_for "$p"
done

But certainly you need to make sure that the jobs running in parallel are not stepping on each other's feet (like writing to the same file).