新手python子进程：“写入错误：管道损坏”
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/4106565/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
newbie python subprocess: "write error: Broken pipe"
提问by mathtick
Thanks to the helpful suggestions below:
感谢以下有用的建议：
So it seems to be fixed when I
所以它似乎是固定的，当我
- separate commands into individual calls to Popen
- stderr=subprocess.PIPE as an argument to each Popen chain.
- 将命令分成对 Popen 的单独调用
- stderr=subprocess.PIPE 作为每个 Popen 链的参数。
The New code:
新代码：
import subprocess
import shlex
import logging
def run_shell_commands(cmds):
""" Run commands and return output from last call to subprocess.Popen.
For usage see the test below.
"""
# split the commands
cmds = cmds.split("|")
cmds = list(map(shlex.split,cmds))
logging.info('%s' % (cmds,))
# run the commands
stdout_old = None
stderr_old = None
p = []
for cmd in cmds:
logging.info('%s' % (cmd,))
p.append(subprocess.Popen(cmd,stdin=stdout_old,stdout=subprocess.PIPE,stderr=subprocess.PIPE))
stdout_old = p[-1].stdout
stderr_old = p[-1].stderr
return p[-1]
pattern = '"^85567 "'
file = "j"
cmd1 = 'grep %s %s | sort -g -k3 | head -10 | cut -d" " -f2,3' % (pattern, file)
p = run_shell_commands(cmd1)
out = p.communicate()
print(out)
Original Post:
原帖：
I've spent too long trying to solve a problem piping a simple subprocess.Popen.
我花了很长时间试图解决一个简单的 subprocess.Popen 管道问题。
Code:
代码：
import subprocess
cmd = 'cat file | sort -g -k3 | head -20 | cut -f2,3' % (pattern,file)
p = subprocess.Popen(cmd,shell=True,stdout=subprocess.PIPE)
for line in p.stdout:
print(line.decode().strip())
Output for file ~1000 lines in length:
文件长度约 1000 行的输出：
...
sort: write failed: standard output: Broken pipe
sort: write error
Output for file >241 lines in length:
文件长度大于 241 行的输出：
...
sort: fflush failed: standard output: Broken pipe
sort: write error
Output for file <241 lines in length is fine.
文件长度 <241 行的输出很好。
I have been reading the docs and googling like mad but there is something fundamental about the subprocess module that I'm missing ... maybe to do with buffers. I've tried p.stdout.flush() and playing with the buffer size and p.wait(). I've tried to reproduce this with commands like 'sleep 20; cat moderatefile' but this seems to run without error.
我一直在阅读文档并疯狂地使用谷歌搜索，但是我缺少关于子流程模块的一些基本知识......也许与缓冲区有关。我试过 p.stdout.flush() 并使用缓冲区大小和 p.wait()。我试图用像'sleep 20; 这样的命令来重现它。cat modemfile' 但这似乎运行没有错误。
采纳答案by Paulo Scardine
From the recipes on subprocessdocs:
从子流程文档的食谱中：
# To replace shell pipeline like output=`dmesg | grep hda`
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
output = p2.communicate()[0]
回答by user225312
You don't need shell=True. Don't invoke the shell. This is how I would do it:
你不需要shell=True. 不要调用外壳。这就是我将如何做到的：
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
stdout_value = p.communicate()[0]
stdout_value # the output
See if you face the problem about the buffer after using this?
看看你用过这个之后有没有遇到过buffer的问题？
回答by mdeous
This is because you shouldn't use "shell pipes" in the command passed to subprocess.Popen, you should use the subprocess.PIPElike this:
这是因为您不应该在传递给 的命令中使用“shell 管道” subprocess.Popen，您应该使用subprocess.PIPE这样的：
from subprocess import Popen, PIPE
p1 = Popen('cat file', stdout=PIPE)
p2 = Popen('sort -g -k 3', stdin=p1.stdout, stdout=PIPE)
p3 = Popen('head -20', stdin=p2.stdout, stdout=PIPE)
p4 = Popen('cut -f2,3', stdin=p3.stdout)
final_output = p4.stdout.read()
But i have to say that what you're trying to do could be done in pure python instead of calling a bunch of shell commands.
但我不得不说，你想要做的事情可以在纯 python 中完成，而不是调用一堆 shell 命令。
回答by Corey Goldberg
try using communicate(), rather than reading directly from stdout.
尝试使用communication()，而不是直接从stdout 读取。
the python docs say this:
python 文档是这样说的：
"Warning Use communicate() rather than .stdin.write, .stdout.read or .stderr.read to avoid deadlocks due to any of the other OS pipe buffers filling up and blocking the child process."
“警告使用communication() 而不是.stdin.write、.stdout.read 或.stderr.read 来避免由于任何其他操作系统管道缓冲区填满并阻塞子进程而导致的死锁。”
http://docs.python.org/library/subprocess.html#subprocess.Popen.stdout
http://docs.python.org/library/subprocess.html#subprocess.Popen.stdout
p = subprocess.Popen(cmd, stdout=subprocess.PIPE)
output = p.communicate[0]
for line in output:
# do stuff
回答by Chris Beecroft
I have been having the same error. Even put the pipe in a bash script and executed that instead of the pipe in Python. From Python it would get the broken pipe error, from bash it wouldn't.
我一直有同样的错误。甚至将管道放在 bash 脚本中并执行它而不是 Python 中的管道。从 Python 中它会得到管道损坏的错误，从 bash 中它不会。
It seems to me that perhaps the last command prior to the head is throwing an error as it's (the sort) STDOUT is closed. Python must be picking up on this whereas with the shell the error is silent. I've changed my code to consume the entire input and the error went away.
在我看来，可能是 head 之前的最后一个命令抛出错误，因为它的（排序）STDOUT 已关闭。Python 必须注意到这一点，而在 shell 中，错误是无声的。我已经更改了我的代码以使用整个输入并且错误消失了。
Would make sense also with smaller files working as the pipe probably buffers the entire output before head exits. This would explain the breaks on larger files.
对于较小的文件也有意义，因为管道可能会在 head 退出之前缓冲整个输出。这将解释较大文件的中断。
e.g., instead of a 'head -1' (in my case, I was only wanting the first line), I did an awk 'NR == 1'
例如，不是'head -1'（在我的情况下，我只想要第一行），我做了一个 awk 'NR == 1'
There are probably better ways of doing this depending on where the 'head -X' occurs in the pipe.
可能有更好的方法来执行此操作，具体取决于管道中出现“head -X”的位置。
