Python 如何从一个 subprocess.Popen 命令同步运行多个命令？

How to run multiple commands synchronously from one subprocess.Popen command?

提问by fredrik

Is it possible to execute an arbitrary number of commands in sequence using the same subprocess command?

是否可以使用相同的子进程命令按顺序执行任意数量的命令？

I need each command to wait for the previous one to complete before executing and I need them all to be executed in the same session/shell. I also need this to work in Python 2.6, Python 3.5. I also need the subprocess command to work in Linux, Windows and macOS (which is why I'm just using echocommands as examples here).

我需要每个命令在执行之前等待前一个命令完成，并且我需要它们都在同一个会话/shell 中执行。我还需要它在 Python 2.6、Python 3.5 中工作。我还需要 subprocess 命令在 Linux、Windows 和 macOS 中工作（这就是我在echo这里仅使用命令作为示例的原因）。

See non-workingcode below:

请参阅下面的非工作代码：

import sys
import subprocess

cmds = ['echo start', 'echo mid', 'echo end']

p = subprocess.Popen(cmd=tuple([item for item in cmds]),
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

for line in iter(p.stdout.readline, b''):
    sys.stdout.flush()
    print(">>> " + line.rstrip())

If this is not possible, which approach should I take in order to execute my commands in synchronous sequence within the same session/shell?

如果这是不可能的，我应该采取哪种方法来在同一个会话/shell 中以同步顺序执行我的命令？

回答by Serge Ballesta

If you want to execute many commands one after the other in the same session/shell, you must start a shell and feed it with all the commands, one at a time followed by a new line, and close the pipe at the end. It makes sense if some commands are not true processes but shell commands that could for example change the shell environment.

如果你想在同一个session/shell 中一个接一个地执行许多命令，你必须启动一个 shell 并提供所有命令，一次一个，然后一个新行，最后关闭管道。如果某些命令不是真正的进程而是可以例如更改 shell 环境的 shell 命令，那么这是有道理的。

Example using Python 2.7 under Windows:

在 Windows 下使用 Python 2.7 的示例：

encoding = 'latin1'
p = subprocess.Popen('cmd.exe', stdin=subprocess.PIPE,
             stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for cmd in cmds:
    p.stdin.write(cmd + "\n")
p.stdin.close()
print p.stdout.read()

To have this code run under Linux, you would have to replace cmd.exewith /bin/bashand probably change the encoding to utf8.

要在Linux下这个代码运行，你就必须更换cmd.exe与/bin/bash和可能更改编码为utf8。

For Python 3, you would have to encode the commands and probably decode their output, and to use parentheses with print.

对于 Python 3，您必须对命令进行编码并可能对它们的输出进行解码，并在打印时使用括号。

Beware: this can only work for little output. If there was enough output to fill the pipe buffer before closing the stdin pipe, this code would deadlock. A more robust way would be to have a second thread to read the output of the commands to avoid that problem.

当心：这只适用于很少的输出。如果在关闭 stdin 管道之前有足够的输出来填充管道缓冲区，则此代码将死锁。更可靠的方法是让第二个线程读取命令的输出以避免该问题。

回答by BuvinJ

This is similar to the answer posted by Serge Ballesta, but not quite. Use his for asynchronous execution, where you don't care about the results. Use mine for synchronous processing and result gathering. Like his answer, I'm showing the Windows solution here - run a bash process in Linux rather than cmd in Windows.

这类似于 Serge Ballesta 发布的答案，但不完全相同。使用他的异步执行，你不关心结果。使用我的进行同步处理和结果收集。就像他的回答一样，我在这里展示了 Windows 解决方案 - 在 Linux 中运行 bash 进程而不是在 Windows 中运行 cmd。

from subprocess import Popen, PIPE
process = Popen( "cmd.exe", shell=False, universal_newlines=True,
                  stdin=PIPE, stdout=PIPE, stderr=PIPE )                             
out, err = process.communicate( commands ) 

USAGE DETAILS:The commandsargument being passed here to the process.communicatemethod is a newline delimitedstring. If, for example you just read a batch file contents into a string, you could run it this way because it would already have the newlines.

用法详细信息：commands这里传递给process.communicate方法的参数是一个换行符分隔的字符串。例如，如果您只是将批处理文件内容读入字符串，则可以以这种方式运行它，因为它已经有了换行符。

Important: your string must end in a newline "\n". If it does not, that final command will fail to execute. Just like if you typed it into your command prompt but didn't hit enterat the end. You will however see a mysterious More?line in the end of the stdout returned. (that's the cause if you encounter this).

重要提示：您的字符串必须以换行符结尾"\n"。如果没有，则最终命令将无法执行。就像你在命令提示符中输入它但最后没有点击一样enter。但是，您会More?在返回的标准输出的末尾看到一条神秘的线。（如果您遇到这种情况，这就是原因）。

process.communicateruns synchronously by definition, and returns the stdout and stderr messages (if you directed them to subprocess.PIPEin your Popen constructor).

process.communicate根据定义同步运行，并返回 stdout 和 stderr 消息（如果您subprocess.PIPE在 Popen 构造函数中将它们定向到）。

When you create a cmd.exeprocess in this way, and pass a string to it, the results will be exactly like if you were to open a command prompt window and entered commands into. And I mean that quite literally. If you test this, you will see that the stdout which is returned contains your commands. (It does NOT matter if you include an @echo offlike if executing a batch file).

当您cmd.exe以这种方式创建一个进程，并将一个字符串传递给它时，结果将与您打开命令提示符窗口并输入命令完全一样。我的意思是字面意思。如果您对此进行测试，您将看到返回的标准输出包含您的命令。（如果您@echo off在执行批处理文件时包含类似内容，则无关紧要）。

Tips for those who care about "clean" stdout results:

给那些关心“干净”标准输出结果的人的提示：

• @echo offwill not suppress your commands from appearing in this returned string, but it does remove extra newlines that find their way in there otherwise. (universal_newlines=True strips an another set of those)

• Including an @symbol prefix to your commands allows them to still execute. In a "normal" batch process that's the line-by-line way to "hide" your commands. In this context, it's a safe an easy marker by which you can find stdout lines you want to remove. (if one were so inclined)

• The cmd.exe "header" will appear in your output (which says the version of Windows etc.). Since you probably want to start your set of commands with @echo off, to cut out the extra newlines, that is also a great way to find where the header lines stopped and your commands/results began.

• @echo off不会禁止您的命令出现在这个返回的字符串中，但它会删除额外的换行符，否则它们会在那里找到。（universal_newlines=True 剥离另一组那些）

• @在命令中包含符号前缀允许它们仍然执行。在“正常”批处理中，这是“隐藏”命令的逐行方式。在这种情况下，它是一个安全的简单标记，您可以通过它找到要删除的标准输出行。（如果有人这么倾向的话）

• cmd.exe“标题”将出现在您的输出中（表示 Windows 的版本等）。由于您可能希望以 , 开始您的命令集@echo off，以删除额外的换行符，这也是查找标题行停止位置和命令/结果开始位置的好方法。

Finally, to address concerns about "large" output filling the pipes and causing you problems - first I think you need a HUGE amount of data coming back for that to be an issue - more than most people will encounter in their use cases. Second, if it really is a concern just open a file for writing and pass that file handle (the reference to the file object) to stdout/err instead of PIPE. Then, do whatever you want with the file you've created.

最后，为了解决对“大”输出填充管道并导致问题的担忧 - 首先，我认为您需要返回大量数据才能成为问题 - 比大多数人在他们的用例中遇到的要多。其次，如果真的是一个问题，只需打开一个文件进行写入并将该文件句柄（对文件对象的引用）传递给 stdout/err 而不是PIPE. 然后，对您创建的文件执行任何您想要的操作。

回答by AlokThakur

One possible solution, looks like its running in same shell:

一种可能的解决方案，看起来像是在同一个 shell 中运行：

subprocess.Popen('echo start;echo mid;echo end', shell=True)

Note - If you pass your command as a string then shell has to be True Note - This is working on linux only, you may have to find something similar way out on windows.

注意 - 如果您将命令作为字符串传递，则 shell 必须为 True 注意 - 这仅适用于 linux，您可能需要在 Windows 上找到类似的方法。

Hope it will help.

希望它会有所帮助。

From python doc -

来自 python 文档 -

On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt.

在 shell=True 的 Unix 上，shell 默认为 /bin/sh。如果 args 是字符串，则该字符串指定要通过 shell 执行的命令。这意味着字符串的格式必须与在 shell 提示符下键入时完全相同。

回答by Marichyasana

Here is a function (and main to run it) that I use. I would say that you can use it for your problem. And it is flexible.

这是我使用的一个函数（和运行它的主要函数）。我会说你可以用它来解决你的问题。而且它很灵活。

# processJobsInAList.py
# 2016-09-27   7:00:00 AM   Central Daylight Time 

import win32process, win32event

def CreateMyProcess2(cmd):
    ''' create process width no window that runs a command with arguments
    and returns the process handle'''
    si   = win32process.STARTUPINFO()
    info = win32process.CreateProcess(
        None,      # AppName
        cmd,       # Command line
        None,      # Process Security
        None,      # Thread Security
        0,         # inherit Handles?
        win32process.NORMAL_PRIORITY_CLASS,
        None,      # New environment
        None,      # Current directory
        si)        # startup info
    # info is tuple (hProcess, hThread, processId, threadId)
    return info[0]

if __name__ == '__main__' :
    ''' create/run a process for each list element in "cmds"
    output may be out of order because processes run concurrently '''

    cmds=["echo my","echo heart","echo belongs","echo to","echo daddy"]
    handles    = []
    for i in range(len(cmds)):
        cmd    = 'cmd /c ' + cmds[i]
        handle = CreateMyProcess2(cmd)
        handles.append(handle)

    rc = win32event.WaitForMultipleObjects( handles, 1, -1)  # 1 wait for all, -1 wait infinite
    print 'return code ',rc

output:
heart
my
belongs
to
daddy
return code 0

输出：
心脏
我
是属于
对
爸爸
返回码为0

UPDATE: If you want to run the same process, which will serialize things for you:
1) Remove line: handles.append(handle)
2) Substitute the variable "handle" in place of the list "handles" on the "WaitFor" line
3) Substitute WaitForSingleObject in place of WaitForMultipleObjects

更新：如果你想运行相同的进程，它会为你序列化：
1）删除行：handles.append(handle)
2）用变量“handle”代替“WaitFor”上的列表“handles”第
3行）用 WaitForSingleObject 代替 WaitForMultipleObjects

回答by Riccardo Petraglia

This one works in python 2.7 and should work also in windows. Probably some small refinement is needed for python >3.

这个适用于python 2.7，也应该适用于Windows。python > 3可能需要一些小的改进。

The produced output is (using date and sleep it is easy to see that the commands are executed in row):

产生的输出是（使用 date 和 sleep 很容易看出命令是按行执行的）：

>>>Die Sep 27 12:47:52 CEST 2016
>>>
>>>Die Sep 27 12:47:54 CEST 2016

As you see the commands are executed in a row.

如您所见，命令是连续执行的。

    import sys
    import subprocess
    import shlex

    cmds = ['date', 'sleep 2', 'date']

    cmds = [shlex.split(x) for x in cmds]

    outputs =[]
    for cmd in cmds:
        outputs.append(subprocess.Popen(cmd,
                                   stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate())


    for line in outputs:
        print ">>>" + line[0].strip()

This is what I obtain merging with @Marichyasana answer:

这是我与@Marichyasana 合并后得到的答案：

import sys
import os


def run_win_cmds(cmds):

    @Marichyasana code (+/-)

def run_unix_cmds(cmds):

    import subprocess
    import shlex


    cmds = [shlex.split(x) for x in cmds]

    outputs =[]
    for cmd in cmds:
        outputs.append(subprocess.Popen(cmd,
                                        stdout=subprocess.PIPE, stderr=subprocess.STDOUT).communicate())


    rc = ''
    for line in outputs:
        rc +=  line[0].strip()+'\n'

    return rc


cmds = ['date', 'sleep 2', 'date']

if os.name == 'nt':
     run_win_cmds(cmds)
elif os.name == 'posix':
    run_unix_cmds(cmds)

Ask is this one do not fit your needs! ;)

问是不是这个不符合您的需求！;)