bash 通过 sbatch 传递命令行参数

Pass command line arguments via sbatch

提问by Jason

Suppose that I have the following simple bash script which I want to submit to a batch server through SLURM:

假设我有以下简单的 bash 脚本，我想通过 SLURM 将其提交到批处理服务器：

#!/bin/bash

#SBATCH -o "outFile"".txt"
#SBATCH -e "errFile"".txt"

hostname

exit 0

In this script, I simply want to write the output of hostnameon a textfile whose full name I control via the command-line, like so:

在这个脚本中，我只想hostname在我通过命令行控制全名的文本文件上写入输出，如下所示：

login-2:jobs$ sbatch -D `pwd` exampleJob.sh 1
Submitted batch job 203775

Unfortunately, it seems that my last command-line argument (1) is not parsed through sbatch, since the files created do not have the suffix I'm looking for and the string "$1" is interpreted literally:

不幸的是，我的最后一个命令行参数 (1) 似乎没有通过 sbatch 解析，因为创建的文件没有我正在寻找的后缀，并且字符串“$1”按字面解释：

login-2:jobs$ ls
errFile.txt  exampleJob.sh outFile.txt

I've looked around places in SOand elsewhere, but I haven't had any luck. Essentially what I'm looking for is the equivalent of the -vswitch of the qsubutility in Torque-enabled clusters.

我环顾了SO和其他地方的地方，但我没有任何运气。本质上，我正在寻找的是相当于启用 Torque 的集群中实用程序的-v开关qsub。

Edit: As mentioned in the underlying comment thread, I solved my problem the hard way: instead of having one single script that would be submitted several times to the batch server, each with different command line arguments, I created a "master script" that simply echoed and redirected the same content onto different scripts, the content of each being changed by the command line parameter passed. Then I submitted all of those to my batch server through sbatch. However, this does not answer the original question, so I hesitate to add it as an answer to my question or mark this question solved.

编辑：正如在底层评论线程中提到的，我用困难的方式解决了我的问题：我没有将一个脚本多次提交给批处理服务器，每个脚本都有不同的命令行参数，而是创建了一个“主脚本”简单地将相同的内容回显并重定向到不同的脚本上，每个脚本的内容都由传递的命令行参数更改。然后我通过sbatch. 但是，这并没有回答原始问题，因此我犹豫是否将其添加为我的问题的答案或将此问题标记为已解决。

采纳答案by Walter A

The lines starting with #SBATCH are not interpreted by bash but are replaced with code by sbatch. The sbatch options do not support $1 vars (only %j and some others, replacing $1 by %1 will not work). When you don't have different sbatch processes running in parallel, you could try

以#SBATCH 开头的行不被bash 解释，而是被sbatch 替换为代码。sbatch 选项不支持 $1 vars（仅 %j 和其他一些，用 %1 替换 $1 将不起作用）。当您没有并行运行的不同 sbatch 进程时，您可以尝试

#!/bin/bash

touch outFile.txt errFile.txt
rm link_out.sbatch link_err.sbatch 2>/dev/null # remove links from previous runs
ln -s outFile.txt link_out.sbatch
ln -s errFile.txt link_err.sbatch

#SBATCH -o link_out.sbatch
#SBATCH -e link_err.sbatch

hostname
# I do not know about the background processing of sbatch, are the jobs still running
# at this point? When they are, you can not delete the temporary symlinks yet.

exit 0

Alternative: As you said in a comment yourself, you could make a masterscript. This script can contain lines like

替代方案：正如您自己在评论中所说，您可以制作一个主脚本。这个脚本可以包含像这样的行

cat  exampleJob.sh.template | sed -e 's/File.txt/File''.txt/' > exampleJob.sh
# I do not know, is the following needed with sbatch?
chmod +x exampleJob.sh

In your template the #SBATCH lines look like

在您的模板中，#SBATCH 行看起来像

#SBATCH -o "outFile.txt"
#SBATCH -e "errFile.txt"

回答by MasterHD

I thought I'd offer some insight because I was also looking for the replacement to the -voption in qsub, which for sbatchcan be accomplished using the --exportoption. I found a nice site herethat shows a list of conversions from Torque to Slurm, and it made the transition muchsmoother.

我想我会提供一些见解，因为我也在寻找 中-v选项的替代品qsub，这sbatch可以使用该--export选项来完成。我在这里找到了一个不错的网站，其中显示了从 Torque 到 Slurm 的转换列表，它使转换更加顺畅。

You can specify the environment variable ahead of time in your bash script:

您可以在 bash 脚本中提前指定环境变量：

$ var_name='1'
$ sbatch -D `pwd` exampleJob.sh --export=var_name

Or define it directly within the sbatchcommand just like qsuballowed:

或者sbatch像qsub允许一样直接在命令中定义它：

$ sbatch -D `pwd` exampleJob.sh --export=var_name='1'

Whether this works in the #preprocessors of exampleJob.shis also another question, but I assume that it should give the same functionality found in Torque.

这是否适用于 的#预处理器exampleJob.sh也是另一个问题，但我认为它应该提供与 Torque 相同的功能。

回答by irritable_phd_syndrom

If you pass your commands via the command line, you can actually bypass the issue of not being able to pass command line arguments in the batch script. So for instance, at the command line :

如果通过命令行传递命令，实际上可以绕过无法在批处理脚本中传递命令行参数的问题。例如，在命令行中：

var1="my_error_file.txt"
var2="my_output_file.txt"
sbatch --error=$var1 --output=$var2 batch_script.sh

回答by PouyaB

Using a wrapper is more convenient. I found this solution from this thread.

使用包装器更方便。我从这个线程找到了这个解决方案。

Basically the problem is that the SBATCH directives are seen as comments by the shell and therefore you can't use the passed arguments in them. Instead you can use a here document to feed in your bash script after the arguments are set accordingly.

基本上问题在于 SBATCH 指令被 shell 视为注释，因此您不能在其中使用传递的参数。相反，您可以在相应地设置参数后使用 here 文档来输入您的 bash 脚本。

In case of your question you can substitute the shell script file with this:

如果您有问题，您可以用以下内容替换 shell 脚本文件：

#!/bin/bash
sbatch <<EOT
#!/bin/bash

#SBATCH -o "outFile"".txt"
#SBATCH -e "errFile"".txt"

hostname

exit 0
EOT

And you run the shell script like this:

然后像这样运行 shell 脚本：

bash [script_name].sh [suffix]

And the outputs will be saved to outFile[suffix].txt and errFile[suffix].txt

并且输出将保存到 outFile[suffix].txt 和 errFile[suffix].txt

回答by user3665696

Something like this works for me and Torque

像这样的东西对我和 Torque 有用

echo "$(pwd)/slurm.qsub 1" | qsub -S /bin/bash -N Slurm-TEST
slurm.qsub:

#!/bin/bash
hostname > outFile.txt 2>errFile.txt
exit 0

回答by Yacine Mahdid

This is an old question but I just stumbled into the same task and I think this solution is simpler:

这是一个老问题，但我偶然发现了同样的任务，我认为这个解决方案更简单：

Let's say I have the variable $OUT_PATHin the bash script launch_analysis.bashand I want to pass this variable to task_0_generate_features.slwhich is my SLURM file to send the computation to a batch server. I would have the following in launch_analysis.bash:

假设我$OUT_PATH在 bash 脚本中有这个变量，launch_analysis.bash我想将此变量传递给task_0_generate_features.sl我的 SLURM 文件，以将计算发送到批处理服务器。我会有以下内容launch_analysis.bash：

`sbatch --export=OUT_PATH=$OUT_PATH task_0_generate_features.sl`

Which is directly accessible in task_0_generate_features.sl

哪个可以直接访问 task_0_generate_features.sl

In @Jason case we would have:

在@Jason 的情况下，我们将有：

sbatch -D `pwd` --export=hostname=$hostname exampleJob.sh

Reference: Using Variables in SLURM Jobs

参考：在 SLURM 作业中使用变量