bash python中子进程的替代方法

Alternative in python to subprocess

提问by Akshya11235

I am trying to write a script which has to make a lot of calls to some bash commands, parse and process the outputs and finally give some output.

我正在尝试编写一个脚本，该脚本必须对一些 bash 命令进行大量调用，解析和处理输出，最后给出一些输出。

I was using subprocess.Popen and subprocess.call

我正在使用 subprocess.Popen 和 subprocess.call

If I understand correct these methods spawn a bah process, run the command, get the output and then kill the process.

如果我理解正确，这些方法会产生一个 bah 进程，运行命令，获取输出，然后终止该进程。

Is there a way to have a bash process running in the background continuously and then the python calls could just go directly to that process? This would be something like bash running as a server and python calls going to it.

有没有办法让 bash 进程在后台连续运行，然后 python 调用可以直接进入该进程？这就像 bash 作为服务器运行并且 python 调用去它一样。

I feel this would optimize the calls a bit as there is no bash process setup and teardown. Or will it give no performance advantage?

我觉得这会优化调用，因为没有 bash 进程设置和拆卸。或者它不会带来任何性能优势？

采纳答案by Maxim Egorushkin

If I understand correct these methods spawn a bah process, run the command, get the output and then kill the process.

如果我理解正确，这些方法会产生一个 bah 进程，运行命令，获取输出，然后终止该进程。

subprocess.Popenis a bit more involved. It actually creates an I/O thread to avoid deadlocks. See https://www.python.org/dev/peps/pep-0324/:

subprocess.Popen涉及更多。它实际上创建了一个 I/O 线程以避免死锁。见https://www.python.org/dev/peps/pep-0324/：

A communicate()method, which makes it easy to send stdindata and read stdoutand stderrdata, without risking deadlocks. Most people are aware of the flow control issues involved with child process communication, but not all have the patience or skills to write a fully correct and deadlock-free select loop. This means that many Python applications contain race conditions. A communicate()method in the standard library solves this problem.

一个communicate()方法，它可以很容易地发送stdin数据和读取stdout和stderr数据，而不用担心死锁。大多数人都知道涉及子进程通信的流量控制问题，但并非所有人都有耐心或技巧来编写完全正确且无死锁的选择循环。这意味着许多 Python 应用程序包含竞争条件。communicate()标准库中的一个方法解决了这个问题。

---

---

Is there a way to have a bash process running in the background continuously and then the python calls could just go directly to that process?

有没有办法让 bash 进程在后台连续运行，然后 python 调用可以直接进入该进程？

Sure, you can still use subprocess.Popenand send messages to you subprocess and receive messages back without terminating the subprocess. In the simplest case your messages can be lines.

当然，您仍然可以在subprocess.Popen不终止子进程的情况下使用并向子进程发送消息并接收消息。在最简单的情况下，您的消息可以是行。

This allows for request-response style protocols as well as publish-subscribe when the subprocess can keep sending you messages back when an event of interest happens.

当子进程可以在感兴趣的事件发生时继续向您发送消息时，这允许请求-响应样式协议以及发布-订阅。

回答by jfs

I feel this would optimize the calls a bit as there is no bash process setup and teardown.

我觉得这会优化调用，因为没有 bash 进程设置和拆卸。

subprocessnever runs the shell unless you ask it explicitly e.g.,

subprocess除非您明确询问，否则永远不会运行 shell，例如，

#!/usr/bin/env python
import subprocess

subprocess.check_call(['ls', '-l'])

This call runs lsprogram without invoking /bin/sh.

此调用运行ls程序而不调用/bin/sh.

Or will it give no performance advantage?

或者它不会带来任何性能优势？

If your subprocess calls actually use the shell e.g., to specify a pipeline consicelyor you use bash process substitutionthat could be verbose and error-prone to define using subprocessmodule directly then it is unlikely that invoking bashis a performance bottleneck-- measure it first.

如果您的子进程调用实际上使用 shell，例如，指定一个管道，或者您使用可能冗长且容易出错的bash 进程替换来subprocess直接定义 using模块，那么调用不太可能bash是性能瓶颈——首先测量它。

There are Python packages that too allow to specify such commands consicely e.g., plumbumcould be used to emulate a shell pipeline.

有一些 Python 包也允许指定此类命令，例如，plumbum可用于模拟 shell 管道。

If you want to use bashas a server process then pexpectis useful for dialog-based interactions with an external process -- though it is unlikely that it affects time performance. fabricallows to run both local and remote commands (ssh).

如果您想bash用作服务器进程，那么pexpect对于与外部进程的基于对话的交互很有用——尽管它不太可能影响时间性能。fabric允许运行本地和远程命令 ( ssh)。

There are other subprocess wrappers such as sargewhich can parse a pipeline specified in a string without invoking the shell e.g., it enables cross-platform support for bash-like syntax (&&, ||, &in command lines)or sh-- a complete subprocessreplacement on Unix that provides TTY by default (it seems full-featured but the shell-like piping is less straightforward). You can even use Python-ish BASHwards-looking syntax to run commands with xonshshell. Again, it is unlikely that it affects performance in a meaningful way in most cases.

还有其他子进程包装器，例如sarge可以在不调用 shell 的情况下解析字符串中指定的管道，例如，它可以跨平台支持类似 bash 的语法（&&, ||,&在命令行中）或sh-- subprocessUnix 上的完整替代品，提供默认情况下 TTY（它看起来功能齐全，但类似外壳的管道不太直接）。您甚至可以使用类似 Python 的 BASHwards 语法来使用xonshshell运行命令。同样，在大多数情况下，它不太可能以有意义的方式影响性能。

The problem of starting and communicating with external processes in a portable manner is complex -- the interaction between processes, pipes, ttys, signals, threading, async. IO, buffering in various places has rough edges. Introducing a new package may complicate things if you don't know how a specific package solve numerous issues related to running shell commands.

以可移植的方式启动和与外部进程通信的问题很复杂——进程、管道、tty、信号、线程、异步之间的交互。IO，缓冲在各个地方都有毛边。如果您不知道特定包如何解决与运行 shell 命令相关的众多问题，那么引入新包可能会使事情复杂化。