bash 在两个命令之间的管道中添加一个大缓冲区
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/8554568/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Add a big buffer to a pipe between two commands
提问by Alex Krauss
Given a bash command line of the form
给定形式的 bash 命令行
commandA | commandB
I want to add a buffer of size ~1MB that sits between commandAand commandB.
I would expect to be able to do this with something of the form
我想添加一个大小为 ~1MB 的缓冲区，它位于commandA和之间commandB。我希望能够用某种形式来做到这一点
commandA | BUFFER | commandB
but what is the command to use for BUFFER?
但是要使用的命令是什么BUFFER？
Remark: I want to do this in order to decouple the two commands to make them parallelize better. The problem is that commandBprocesses data in large chunks, which currently means that commandAblocks until commandBis done with a chunk. So everything runs sequentially :-(
备注：我想这样做是为了解耦这两个命令，使它们更好地并行化。问题在于commandB以大块的形式处理数据，这目前意味着commandA阻塞直到commandB处理完一个块。所以一切都按顺序运行:-(
回答by Eugen Rieck
BUFFER is called buffer. (man 1 buffer, maybe after apt-get install buffer)
BUFFER 称为缓冲区。（人 1 缓冲区，也许在 apt-get 安装缓冲区之后）
回答by Johannes Gerer
There is another tool, pv- pipe viewer:
还有另一个工具，pv-管道查看器：
process1 | pv -pterbTCB 1G | process2
Bspecifies the buffer size, here 1 GigibyteCdisablessplice, which is required forBTshows the buffer levelpterbare the default display switches needed due to the presence ofT
B指定缓冲区大小，这里是 1 GigibyteCdisablessplice，这是必需的BT显示缓冲液位pterb是由于存在而需要的默认显示开关T
pvmight be available on systems where mbuffer/bufferis not in the official repositories (such as arch linux).
pv可能在mbuffer/buffer不在官方存储库中的系统上可用（例如arch linux）。
回答by sehe
You can use
您可以使用
- buffer (mentioned)
- mbuffer (works on solaris too, possibly other UNIXes)
- 缓冲区（提及）
- mbuffer（也适用于 solaris，也可能适用于其他 UNIX）
E.g.
例如
process1 | mbuffer -m 1024M | process2
to use a 1G buffer
使用 1G 缓冲区
回答by ceving
The program bufferuses shared memory. This might be a problem, because in case of an error, memory may leak, because shared memory can outlive the program, which allocated the memory.
该程序buffer使用共享内存。这可能是一个问题，因为在发生错误的情况下，内存可能会泄漏，因为共享内存可能比分配内存的程序寿命更长。
An alternative may be GNU dd:
另一种选择可能是 GNU dd：
commandA |
dd status=none iflag=fullblock bs=1M |
commandB
It is important to use the fullblockoption. Otherwise ddmay cause data loss, when reading from a pipe.
使用该fullblock选项很重要。否则dd从管道读取时可能会导致数据丢失。
Parameters of ddexplained
参数dd说明
status=none
Set the level of information to print to stderr; 'none' suppresses everything but error messages
iflag=fullblock
accumulate full blocks of input
bs=1M
read and write up to one Mega bytes at a time (default: 512 bytes);
状态=无
设置要打印到stderr的信息级别；'none' 抑制除错误消息之外的所有内容
iflag=fullblock
累积完整的输入块
bs=1M
一次最多读取和写入 1 兆字节（默认值：512 字节）；
回答by Samus_
alternatively you could use a named pipe and run them in parallel:
或者，您可以使用命名管道并并行运行它们：
mkfifo myfifo
commandB < myfifo &
commandA > myfifo
rm myfifo
