等待脚本中的 bash 后台作业完成

Wait for bash background jobs in script to be finished

提问by mark

To maximize CPU usage (I run things on a Debian Lenny in EC2) I have a simple script to launch jobs in parallel:

为了最大限度地提高 CPU 使用率（我在 EC2 中的 Debian Lenny 上运行），我有一个简单的脚本来并行启动作业：

#!/bin/bash

for i in apache-200901*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200902*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200903*.log; do echo "Processing $i ..."; do_something_important; done &
for i in apache-200904*.log; do echo "Processing $i ..."; do_something_important; done &
...

I'm quite satisfied with this working solution, however I couldn't figure out how to write further code which only executed once all of the loops have been completed.

我对这个工作解决方案非常满意，但是我不知道如何编写更多代码，这些代码仅在所有循环完成后才执行。

Is there a way to get control of this?

有没有办法控制这个？

回答by eduffy

There's a bashbuiltin command for that.

有一个bash内置命令。

wait [n ...]
      Wait for each specified process and return its termination  sta‐
      tus.   Each  n  may be a process ID or a job specification; if a
      job spec is given, all processes  in  that  job's  pipeline  are
      waited  for.  If n is not given, all currently active child pro‐
      cesses are waited for, and the return  status  is  zero.   If  n
      specifies  a  non-existent  process or job, the return status is
      127.  Otherwise, the return status is the  exit  status  of  the
      last process or job waited for.

回答by Ole Tange

Using GNU Parallel will make your script even shorter and possibly more efficient:

使用 GNU Parallel 将使您的脚本更短，并且可能更高效：

parallel 'echo "Processing "{}" ..."; do_something_important {}' ::: apache-*.log

This will run one job per CPU core and continue to do that until all files are processed.

这将为每个 CPU 内核运行一项作业，并继续执行此操作，直到处理完所有文件。

Your solution will basically split the jobs into groups before running. Here 32 jobs in 4 groups:

您的解决方案基本上会在运行之前将作业分成几组。这里有 4 个组中的 32 个工作：

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel 会在完成后生成一个新进程 - 保持 CPU 处于活动状态，从而节省时间：

To learn more:

了解更多：

• Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• Walk through the tutorial (man parallel_tutorial). You command line will love you for it.

• 观看介绍视频以进行快速介绍：https: //www.youtube.com/playlist?list=PL284C9FF2488BC6D1
• 完成教程 (man parallel_tutorial)。您的命令行会因此而爱您。

回答by Olivier Lalonde

I had to do this recently and ended up with the following solution:

我最近不得不这样做，最终得到了以下解决方案：

while true; do
  wait -n || {
    code="$?"
    ([[ $code = "127" ]] && exit 0 || exit "$code")
    break
  }
done;

Here's how it works:

这是它的工作原理：

wait -nexits as soon as one of the (potentially many) background jobs exits. It always evaluates to true and the loop goes on until:

wait -n一旦（可能很多）后台作业之一退出，它就会退出。它总是评估为真，循环继续直到：

1. Exit code 127: the last background job successfully exited. In that case, we ignore the exit code and exit the sub-shell with code 0.
2. Any of the background job failed. We just exit the sub-shell with that exit code.

1. 退出代码127：上一个后台作业成功退出。在这种情况下，我们忽略退出代码并使用代码 0 退出子 shell。
2. 任何后台作业失败。我们只是用那个退出代码退出子 shell。

With set -e, this will guarantee that the script will terminate early and pass through the exit code of any failed background job.

使用set -e，这将保证脚本将提前终止并通过任何失败的后台作业的退出代码。

回答by Radu

This is my crude solution:

这是我的粗略解决方案：

function run_task {
        cmd=
        output=
        concurency=
        if [ -f ${output}.done ]; then
                # experiment already run
                echo "Command already run: $cmd. Found output $output"
                return
        fi
        count=`jobs -p | wc -l`
        echo "New active task #$count:  $cmd > $output"
        $cmd > $output && touch $output.done &
        stop=$(($count >= $concurency))
        while [ $stop -eq 1 ]; do
                echo "Waiting for $count worker threads..."
                sleep 1
                count=`jobs -p | wc -l`
                stop=$(($count > $concurency))
        done
}

The idea is to use "jobs" to see how many children are active in the background and wait till this number drops (a child exits). Once a child exists, the next task can be started.

这个想法是使用“工作”来查看有多少孩子在后台活跃，然后等到这个数字下降（孩子退出）。一旦孩子存在，就可以开始下一个任务。

As you can see, there is also a bit of extra logic to avoid running the same experiments/commands multiple times. It does the job for me.. However, this logic could be either skipped or further improved (e.g., check for file creation timestamps, input parameters, etc.).

如您所见，还有一些额外的逻辑可以避免多次运行相同的实验/命令。它为我完成了这项工作。然而，这个逻辑可以被跳过或进一步改进（例如，检查文件创建时间戳、输入参数等）。