SLURM sbatch job array for the same script but with different input string arguments run in parallel

和自甴很熟 提交于 2020-07-05 12:34:51

问题


My question is similar with this one, and the difference is that my different arguments are not numbers but strings.

If I have a script (myscript.R) that takes two strings as arguments: "text-a", "text-A". My shell script for sbatch would be:

#!/bin/bash

#SBATCH -n 1
#SBATCH -c 12
#SBATCH -t 120:00:00
#SBATCH --partition=main
#SBATCH --export=ALL

srun ./myscript.R "text-a" "text-A"

Now I have a few different input strings that I'd like to run with:

first <- c("text-a","text-b","text-c","text-d")
second <- c("text-A","text-B","text-C","text-D")

and I want to run myscript.R with combinations of the texts, for example:

srun ./myscript.R "text-a" "text-A"
srun ./myscript.R "text-b" "text-B"
srun ./myscript.R "text-c" "text-C"
srun ./myscript.R "text-d" "text-D"

But if I put them in the same shell script, they'll run sequentially. I only know that I can use #SBATCH -a 0-10 when the arguments are index. If I want to submit the four scripts at the same time and each of them with the exact same settings (especially each one need to be assigned -c 12), how can I do that?

Thanks!


回答1:


You can store de list of argument values in an array and use the SLURM_ARRAY_TASK_ID env variable to index that array.

#!/bin/bash

#SBATCH -n 1
#SBATCH -c 12
#SBATCH -t 120:00:00
#SBATCH --partition=main
#SBATCH --export=ALL
#SBATCH --array=0-3

A=(text-{a..d}) # This is equivalent to A=(text-a text-b ... text-d)
B=(text-{A..D})

srun ./myscript.R "${A[$SLURM_ARRAY_TASK_ID]}" "${B[$SLURM_ARRAY_TASK_ID]}"

and simply submit it with sbatch.



来源:https://stackoverflow.com/questions/43642029/slurm-sbatch-job-array-for-the-same-script-but-with-different-input-string-argum

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!