Python 在进程运行时不断打印子进程输出
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/4417546/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Constantly print Subprocess output while process is running
提问by Ingo Fischer
To launch programs from my Python-scripts, I'm using the following method:
要从我的 Python 脚本启动程序，我使用以下方法：
def execute(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
output = process.communicate()[0]
exitCode = process.returncode
if (exitCode == 0):
return output
else:
raise ProcessException(command, exitCode, output)
So when i launch a process like Process.execute("mvn clean install"), my program waits until the the process is finished, and only then i get the complete output of my program. This is annoying if i'm running a process that takes a while to finish.
因此，当我启动类似 的过程时Process.execute("mvn clean install")，我的程序会等到该过程完成，然后才能获得程序的完整输出。如果我正在运行一个需要一段时间才能完成的进程，这很烦人。
Can i let my program write the process output line by line, by polling the process output before it finishes in a loop or something?
我可以让我的程序通过在循环结束之前轮询进程输出来逐行写入进程输出吗？
**[EDIT] Sorry i didn't search very well before posting this question. Threading is actually the key. Found an example here which shows how to do it: ** Python Subprocess.Popen from a thread
**[编辑] 抱歉，在发布此问题之前我没有很好地搜索。线程实际上是关键。在这里找到了一个示例，它展示了如何做到这一点：** Python Subprocess.Popen from a thread
采纳答案by tokland
You can use iterto process lines as soon as the command outputs them: lines = iter(fd.readline, ""). Here's a full example showing a typical use case (thanks to @jfs for helping out):
您可以在命令输出后立即使用iter处理行：lines = iter(fd.readline, ""). 这是一个显示典型用例的完整示例（感谢@jfs 提供帮助）：
from __future__ import print_function # Only Python 2.x
import subprocess
def execute(cmd):
popen = subprocess.Popen(cmd, stdout=subprocess.PIPE, universal_newlines=True)
for stdout_line in iter(popen.stdout.readline, ""):
yield stdout_line
popen.stdout.close()
return_code = popen.wait()
if return_code:
raise subprocess.CalledProcessError(return_code, cmd)
# Example
for path in execute(["locate", "a"]):
print(path, end="")
回答by Ingo Fischer
Ok i managed to solve it without threads (any suggestions why using threads would be better are appreciated) by using a snippet from this question Intercepting stdout of a subprocess while it is running
好的，我设法在没有线程的情况下解决了它（任何建议为什么使用线程会更好，都值得赞赏）通过使用这个问题的片段Intercepting stdout of a subprocess while it is running
def execute(command):
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
# Poll process for new output until finished
while True:
nextline = process.stdout.readline()
if nextline == '' and process.poll() is not None:
break
sys.stdout.write(nextline)
sys.stdout.flush()
output = process.communicate()[0]
exitCode = process.returncode
if (exitCode == 0):
return output
else:
raise ProcessException(command, exitCode, output)
回答by user1379351
For anyone trying the answers to this question to get the stdout from a Python script note that Python buffers its stdout, and therefore it may take a while to see the stdout.
对于尝试回答这个问题以从 Python 脚本中获取标准输出的任何人，请注意 Python 缓冲其标准输出，因此可能需要一段时间才能看到标准输出。
This can be rectified by adding the following after each stdout write in the target script:
这可以通过在目标脚本中的每个 stdout 写入后添加以下内容来纠正：
sys.stdout.flush()
回答by user3759376
@tokland
@托克兰
tried your code and corrected it for 3.4 and windows dir.cmd is a simple dir command, saved as cmd-file
尝试了您的代码并将其更正为 3.4 和 windows dir.cmd 是一个简单的 dir 命令，保存为 cmd-file
import subprocess
c = "dir.cmd"
def execute(command):
popen = subprocess.Popen(command, stdout=subprocess.PIPE,bufsize=1)
lines_iterator = iter(popen.stdout.readline, b"")
while popen.poll() is None:
for line in lines_iterator:
nline = line.rstrip()
print(nline.decode("latin"), end = "\r\n",flush =True) # yield line
execute(c)
回答by jfs
To print subprocess' output line-by-line as soon as its stdout buffer is flushed in Python 3:
要在 Python 3 中刷新其 stdout 缓冲区后，立即逐行打印子进程的输出：
from subprocess import Popen, PIPE, CalledProcessError
with Popen(cmd, stdout=PIPE, bufsize=1, universal_newlines=True) as p:
for line in p.stdout:
print(line, end='') # process line here
if p.returncode != 0:
raise CalledProcessError(p.returncode, p.args)
Notice: you do not need p.poll()-- the loop ends when eof is reached. And you do not need iter(p.stdout.readline, '')-- the read-ahead bug is fixed in Python 3.
注意：您不需要p.poll()——当达到 eof 时循环结束。而且您不需要iter(p.stdout.readline, '')——预读错误已在 Python 3 中修复。
See also, Python: read streaming input from subprocess.communicate().
回答by Will
In case someone wants to read from both stdoutand stderrat the same time using threads, this is what I came up with:
如果有人想从两个读stdout和stderr在同一时间使用线程，这是我想出了：
import threading
import subprocess
import Queue
class AsyncLineReader(threading.Thread):
def __init__(self, fd, outputQueue):
threading.Thread.__init__(self)
assert isinstance(outputQueue, Queue.Queue)
assert callable(fd.readline)
self.fd = fd
self.outputQueue = outputQueue
def run(self):
map(self.outputQueue.put, iter(self.fd.readline, ''))
def eof(self):
return not self.is_alive() and self.outputQueue.empty()
@classmethod
def getForFd(cls, fd, start=True):
queue = Queue.Queue()
reader = cls(fd, queue)
if start:
reader.start()
return reader, queue
process = subprocess.Popen(command, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
(stdoutReader, stdoutQueue) = AsyncLineReader.getForFd(process.stdout)
(stderrReader, stderrQueue) = AsyncLineReader.getForFd(process.stderr)
# Keep checking queues until there is no more output.
while not stdoutReader.eof() or not stderrReader.eof():
# Process all available lines from the stdout Queue.
while not stdoutQueue.empty():
line = stdoutQueue.get()
print 'Received stdout: ' + repr(line)
# Do stuff with stdout line.
# Process all available lines from the stderr Queue.
while not stderrQueue.empty():
line = stderrQueue.get()
print 'Received stderr: ' + repr(line)
# Do stuff with stderr line.
# Sleep for a short time to avoid excessive CPU use while waiting for data.
sleep(0.05)
print "Waiting for async readers to finish..."
stdoutReader.join()
stderrReader.join()
# Close subprocess' file descriptors.
process.stdout.close()
process.stderr.close()
print "Waiting for process to exit..."
returnCode = process.wait()
if returnCode != 0:
raise subprocess.CalledProcessError(returnCode, command)
I just wanted to share this, as I ended up on this question trying to do something similar, but none of the answers solved my problem. Hopefully it helps someone!
我只是想分享这个，因为我最终在这个问题上尝试做类似的事情，但没有一个答案解决了我的问题。希望它可以帮助某人！
Note that in my use case, an external process kills the process that we Popen().
请注意，在我的用例中，外部进程会杀死我们Popen().
回答by Robert Nagtegaal
This PoC constantly reads the output from a process and can be accessed when needed. Only the last result is kept, all other output is discarded, hence prevents the PIPE from growing out of memory:
该 PoC 不断读取进程的输出，并且可以在需要时访问。只保留最后一个结果，丢弃所有其他输出，从而防止 PIPE 内存不足：
import subprocess
import time
import threading
import Queue
class FlushPipe(object):
def __init__(self):
self.command = ['python', './print_date.py']
self.process = None
self.process_output = Queue.LifoQueue(0)
self.capture_output = threading.Thread(target=self.output_reader)
def output_reader(self):
for line in iter(self.process.stdout.readline, b''):
self.process_output.put_nowait(line)
def start_process(self):
self.process = subprocess.Popen(self.command,
stdout=subprocess.PIPE)
self.capture_output.start()
def get_output_for_processing(self):
line = self.process_output.get()
print ">>>" + line
if __name__ == "__main__":
flush_pipe = FlushPipe()
flush_pipe.start_process()
now = time.time()
while time.time() - now < 10:
flush_pipe.get_output_for_processing()
time.sleep(2.5)
flush_pipe.capture_output.join(timeout=0.001)
flush_pipe.process.kill()
print_date.py
打印日期.py
#!/usr/bin/env python
import time
if __name__ == "__main__":
while True:
print str(time.time())
time.sleep(0.01)
output: You can clearly see that there is only output from ~2.5s interval nothing in between.
输出：您可以清楚地看到只有 ~2.5s 间隔的输出，中间没有任何内容。
>>>1520535158.51
>>>1520535161.01
>>>1520535163.51
>>>1520535166.01
回答by arod
This works at least in Python3.4
这至少适用于 Python3.4
import subprocess
process = subprocess.Popen(cmd_list, stdout=subprocess.PIPE)
for line in process.stdout:
print(line.decode().strip())
回答by user7017793
In Python >= 3.5 using subprocess.runworks for me:
在 Python >= 3.5 中使用subprocess.run对我有用：
import subprocess
cmd = 'echo foo; sleep 1; echo foo; sleep 2; echo foo'
subprocess.run(cmd, shell=True)
(getting the output during execution also works without shell=True)
https://docs.python.org/3/library/subprocess.html#subprocess.run
（在执行期间获取输出也可以在没有shell=True）
https://docs.python.org/3/library/subprocess.html#subprocess.run
回答by Alleo
To answer the original question, the best way IMO is just redirecting subprocess stdoutdirectly to your program's stdout(optionally, the same can be done for stderr, as in example below)
要回答原始问题，IMO 的最佳方法是将子进程stdout直接重定向到您的程序stdout（可选，可以对 执行相同的操作stderr，如下例所示）
p = Popen(cmd, stdout=sys.stdout, stderr=sys.stderr)
p.communicate()
