bash 使用 xargs 运行多个命令

Running multiple commands with xargs

提问by Dagang

cat a.txt | xargs -I % echo %

In the example above, xargs takes echo %as the command argument. But in some cases, I need multiple commands to process the argument instead of one. For example:

在上面的例子中， xargsecho %作为命令参数。但在某些情况下，我需要多个命令来处理参数而不是一个。例如：

cat a.txt | xargs -I % {command1; command2; ... }

But xargs doesn't accept this form. One solution I know is that I can define a function to wrap the commands, but it's not a pipeline, I don't prefer it. Is there another solution?

但是 xargs 不接受这种形式。我知道的一个解决方案是我可以定义一个函数来包装命令，但它不是管道，我不喜欢它。还有其他解决方案吗？

回答by Keith Thompson

cat a.txt | xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

...or, without a Useless Use Of cat:

...或者，没有无用的使用 cat：

<a.txt xargs -d $'\n' sh -c 'for arg do command1 "$arg"; command2 "$arg"; ...; done' _

---

---

To explain some of the finer points:

解释一些细节：

• The use of "$arg"instead of %(and the absence of -Iin the xargscommand line) is for security reasons: Passing data on sh's command-line argument list instead of substituting it into code prevents content that data might contain (such as $(rm -rf ~), to take a particularly malicious example) from being executed as code.

• Similarly, the use of -d $'\n'is a GNU extension which causes xargsto treat each line of the input file as a separate data item. Either this or -0(which expects NULs instead of newlines) is necessary to prevent xargs from trying to apply shell-like (but not quiteshell-compatible) parsing to the stream it reads. (If you don't have GNU xargs, you can use tr '\n' '\0' <a.txt | xargs -0 ...to get line-oriented reading without -d).

• The _is a placeholder for $0, such that other data values added by xargsbecome $1and onward, which happens to be the default set of values a forloop iterates over.

• 出于安全原因，使用"$arg"代替%（以及-I在xargs命令行中不存在）是出于安全原因：在sh的命令行参数列表上传递数据而不是将其替换为代码可防止数据可能包含的内容（例如$(rm -rf ~)，采取特别恶意示例）不会被作为代码执行。

• 类似地，使用-d $'\n'是一个 GNU 扩展，它导致xargs将输入文件的每一行视为单独的数据项。这或-0（期望 NUL 而不是换行符）是必要的，以防止 xargs 尝试将类似 shell（但不是完全兼容 shell）的解析应用于它读取的流。（如果您没有 GNU xargs，则可以使用tr '\n' '\0' <a.txt | xargs -0 ...来获得面向行的阅读，而无需-d）。

• The_是 for 的占位符$0，这样其他数据值由xargsbecome$1和 onward添加，这恰好是for循环迭代的默认值集。

回答by Ole Tange

With GNU Parallel you can do:

使用 GNU Parallel，您可以：

cat a.txt | parallel 'command1 {}; command2 {}; ...; '

Watch the intro videos to learn more: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

观看介绍视频以了解更多信息：https: //www.youtube.com/playlist?list=PL284C9FF2488BC6D1

For security reasons it is recommended you use your package manager to install. But if you cannot do that then you can use this 10 seconds installation.

出于安全原因，建议您使用包管理器进行安装。但是如果你不能这样做，那么你可以使用这个 10 秒的安装。

The 10 seconds installation will try to do a full installation; if that fails, a personal installation; if that fails, a minimal installation.

10秒安装会尝试做完整安装；如果失败，个人安装；如果失败，则进行最小安装。

$ (wget -O - pi.dk/3 || lynx -source pi.dk/3 || curl pi.dk/3/ || \
   fetch -o - http://pi.dk/3 ) > install.sh
$ sha1sum install.sh | grep 3374ec53bacb199b245af2dda86df6c9
12345678 3374ec53 bacb199b 245af2dd a86df6c9
$ md5sum install.sh | grep 029a9ac06e8b5bc6052eac57b2c3c9ca
029a9ac0 6e8b5bc6 052eac57 b2c3c9ca
$ sha512sum install.sh | grep f517006d9897747bed8a4694b1acba1b
40f53af6 9e20dae5 713ba06c f517006d 9897747b ed8a4694 b1acba1b 1464beb4
60055629 3f2356f3 3e9c4e3c 76e3f3af a9db4b32 bd33322b 975696fc e6b23cfb
$ bash install.sh

回答by hmontoliu

This is just another approach without xargs nor cat:

这只是另一种没有 xargs 和 cat 的方法：

while read stuff; do
  command1 "$stuff"
  command2 "$stuff"
  ...
done < a.txt

回答by Ossama

You can use

您可以使用

cat file.txt | xargs -i  sh -c 'command {} | command2 {} && command3 {}'

{} = variable for each line on the text file

{} = 文本文件中每一行的变量

回答by mwm

One thing I do is to add to .bashrc/.profile this function:

我做的一件事是将这个函数添加到 .bashrc/.profile 中：

function each() {
    while read line; do
        for f in "$@"; do
            $f $line
        done
    done
}

then you can do things like

然后你可以做这样的事情

... | each command1 command2 "command3 has spaces"

which is less verbose than xargs or -exec. You could also modify the function to insert the value from the read at an arbitrary location in the commands to each, if you needed that behavior also.

这比 xargs 或 -exec 更简洁。如果您还需要该行为，您还可以修改该函数以在命令中的任意位置将读取的值插入到每个中。

回答by brablc

I prefer style which allows dry run mode (without | sh) :

我更喜欢允许试运行模式（没有| sh）的样式：

cat a.txt | xargs -I % echo "command1; command2; ... " | sh

Works with pipes too:

也适用于管道：

cat a.txt | xargs -I % echo "echo % | cat " | sh

回答by sdkks

A little late to the party.

聚会有点晚了。

I use format below for compressing my directories with thousands of tiny files before migrating. If you don't need single quotes inside commands, it should work.

在迁移之前，我使用下面的格式压缩包含数千个小文件的目录。如果您不需要在命令中使用单引号，它应该可以工作。

With some modification, I'm sure it will be useful for someone. Tested in Cygwin(babun)

通过一些修改，我相信它会对某人有用。在Cygwin(babun) 中测试

find . -maxdepth 1 ! -path . -type d -print0 | xargs -0 -I @@ bash -c '{ tar caf "@@.tar.lzop" "@@" && echo Completed compressing directory "@@" ; }'

find .Find here
-maxdepth 1Don't go into child directories
! -path .Exclude . / Current directory path
-type dmatch only directories
-print0Separate output by null bytes \0
| xargsPipe to xargs
-0Input is null separated bytes
-I @@Placeholder is @@. Replace @@ with input.
bash -c '...'Run Bash command
{...}Command grouping
&&Execute next command only if previous command exited successfully (exit 0)

find .在这里查找
-maxdepth 1不要进入子目录
! -path .Exclude 。/ 当前目录路径
-type d只匹配目录
-print0用空字节分隔输出 \0
| xargs管道到 xargs
-0输入是空分隔字节
-I @@占位符是@@。用输入替换@@。
bash -c '...'运行 Bash 命令
{...}命令分组
&&仅当上一个命令成功退出时才执行下一个命令（退出 0）

Final ;is important, otherwise it will fail.

final;很重要，否则会失败。

Output:

输出：

Completed compressing directory ./Directory1 with meta characters in it
Completed compressing directory ./Directory2 with meta characters in it
Completed compressing directory ./Directory3 with meta characters in it

2018 July Update:

2018 年 7 月更新：

If you love hacks and playing around, here is something interesting:

如果你喜欢黑客和玩耍，这里有一些有趣的事情：

echo "a b c" > a.txt
echo "123" >> a.txt
echo "###this is a comment" >> a.txt
cat a.txt
myCommandWithDifferentQuotes=$(cat <<'EOF'                                     
echo "command 1: $@"; echo 'will you do the fandango?'; echo "command 2: $@"; echo
EOF
)
< a.txt xargs -I @@ bash -c "$myCommandWithDifferentQuotes" -- @@

Output:

输出：

command 1: a b c
will you do the fandango?
command 2: a b c

command 1: 123
will you do the fandango?
command 2: 123

command 1: ###this is a comment
will you do the fandango?
command 2: ###this is a comment

Explanation:
-Create a single liner script and store it in a variable
-xargsreads a.txtand executes it as bashscript
-@@makes sure every time an entire line is passed
-Putting @@after --makes sure @@is taken as positional parameter input to bashcommand, not a bashstart OPTION, i.e. like -citself which means run command

说明：
-创建一个单一的衬垫脚本，并将其存储在一个变量
-xargs读取a.txt并执行它作为bash脚本
-@@务必使每一个整行传递时间
-把@@后--确保@@被视为位置参数输入bash命令，而不是一个bash开始OPTION，即喜欢-c它自己，这意味着run command

--is magical, it works with many other things, i.e. ssh, even kubectl

--很神奇，它适用于许多其他事情，即ssh，甚至kubectl

回答by Gert van den Berg

This seems to be the safest version.

这似乎是最安全的版本。

tr '[\n]' '[
tr '[\n]' '[
tr '[\n]' '[
tr '[\n]' '[
-c   If the -c option is present, then  commands  are  read  from  the  first  non-option  argument  com‐
     mand_string.   If there are arguments after the command_string, the first argument is assigned to 
cat a.txt | xargs bash -c 'command1 $@; command2 $@' bash

     and any remaining arguments are assigned to the positional parameters.  The assignment  to  
cat a.txt | xargs -n 5 bash -c 'echo -n `date +%Y%m%d-%H%M%S:` ; echo " data: " $@; echo "data again: " $@' bash
  sets
     the name of the shell, which is used in warning and error messages.
]' < a.txt | xargs -r0 -n10 -P32 /bin/bash -c 'command1 "$@"; command2 "$@";' ''
]' < a.txt | xargs -r0 /bin/bash -c 'for f in "$@"; do command1 "$f"; command2 "$f"; done;' ''
]' < a.txt | xargs -r0 -n1 /bin/bash -c 'command1 "$@"; command2 "$@";' ''
]' < a.txt | xargs -r0 /bin/bash -c 'command1 "$@"; command2 "$@";' ''

(-0can be removed and the trreplaced with a redirect (or the file can be replaced with a null separated file instead). It is mainly in there since I mainly use xargswith findwith -print0output) (This might also be relevant on xargsversions without the -0extension)

（-0可以被删除，tr用重定向替代（或文件可以用零来代替分隔的文件代替），它主要是在那里，因为我主要使用xargs具有find与-print0输出）（这也可能是对相关的xargs版本没有-0扩展名）

It is safe, since args will pass the parameters to the shell as an array when executing it. The shell (at least bash) would then pass them as an unaltered array to the other processes when all are obtained using ["$@"][1]

这是安全的，因为 args 在执行时会将参数作为数组传递给 shell。外壳（至少bash）然后将它们作为未更改的数组传递给其他进程["$@"][1]

If you use ...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' '', the assignment will fail if the string contains double quotes. This is true for every variant using -ior -I. (Due to it being replaced into a string, you can always inject commands by inserting unexpected characters (like quotes, backticks or dollar signs) into the input data)

如果使用...| xargs -r0 -I{} bash -c 'f="{}"; command "$f";' ''，则如果字符串包含双引号，则赋值将失败。对于使用-i或 的每个变体都是如此-I。（由于它被替换为字符串，您始终可以通过在输入数据中插入意外字符（如引号、反引号或美元符号）来注入命令）

If the commands can only take one parameter at a time:

如果命令一次只能接受一个参数：

... | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'

Or with somewhat less processes:

或者使用更少的流程：

 ... | xargs -I % sh -c 'command1; command2; ...'

If you have GNU xargsor another with the -Pextension and you want to run 32 processes in parallel, each with not more than 10 parameters for each command:

如果您有 GNUxargs或其他带有-P扩展名的程序，并且您想并行运行 32 个进程，每个进程的每个命令的参数不超过 10 个：

$ bash 1366 $>  /bin/ls | cat
#Makefile#
#README#
Makefile
README

This should be robust against any special characters in the input. (If the input is null separated.) The trversion will get some invalid input if some of the lines contain newlines, but that is unavoidable with a newline separated file.

这对于输入中的任何特殊字符应该是健壮的。（如果输入是空分隔的。）tr如果某些行包含换行符，版本将获得一些无效输入，但对于换行符分隔的文件，这是不可避免的。

The blank first parameter for bash -cis due to this: (From the bashman page) (Thanks @clacke)

空白的第一个参数bash -c是由于：（来自bash手册页）（感谢@clacke）

$ bash 1367 $>  ls | xargs -n1 -I % sh -i -c 'echo 1 %; echo 2 %'
1
1
1
1 Makefile
2 Makefile
1 README
2 README

回答by tavvit

Another possible solution that works for me is something like -

对我有用的另一个可能的解决方案是 -

$ bash 1368 $>  ls | xargs -n1 -I % perl -e 'system("echo 1 %"); system("echo 2 %");'
1 #Makefile#
2 #Makefile#
1 #README#
2 #README#
1 Makefile
2 Makefile
1 README
2 README
$ bash 1369 $>  

Note the 'bash' at the end - I assume it is passed as argv[0] to bash. Without it in this syntax the first parameter to each command is lost. It may be any word.

注意最后的 'bash' - 我假设它是作为 argv[0] 传递给 bash 的。如果没有这个语法，每个命令的第一个参数就会丢失。它可以是任何词。

Example:

例子：

##代码##

回答by Krazy Glew

My current BKM for this is

我目前的 BKM 是

##代码##

It is unfortunate that this uses perl, which is less likely to be installed than bash; but it handles more input that the accepted answer. (I welcome a ubiquitous version that does not rely on perl.)

不幸的是，这里使用了perl，它比bash 安装的可能性更小；但它处理的输入比接受的答案要多。（我欢迎一个不依赖 perl 的无处不在的版本。）

@KeithThompson's suggestion of

@KeithThompson 的建议

##代码##

is great - unless you have the shell comment character # in your input, in which case part of the first command and all of the second command will be truncated.

很棒 - 除非您的输入中有 shell 注释字符 #，在这种情况下，第一个命令的一部分和第二个命令的所有部分都将被截断。

Hashes # can be quite common, if the input is derived from a filesystem listing, such as ls or find, and your editor creates temporary files with # in their name.

如果输入来自文件系统列表（例如 ls 或 find），并且您的编辑器创建名称中带有 # 的临时文件，则哈希 # 可能非常常见。

Example of the problem:

问题示例：

##代码##

Oops, here is the problem:

哎呀，问题来了：

##代码##

Ahh, that's better:

啊，这样更好：

##代码##