在 Python 中，我如何知道进程何时完成？

In Python, how do I know when a process is finished?

提问by Matthew Walker

From within a Python GUI (PyGTK) I start a process (using multiprocessing). The process takes a long time (~20 minutes) to finish. When the process is finished I would like to clean it up (extract the results and join the process). How do I know when the process has finished?

在 Python GUI (PyGTK) 中，我启动了一个进程（使用多处理）。该过程需要很长时间（约 20 分钟）才能完成。过程完成后，我想清理它（提取结果并加入过程）。我如何知道该过程何时完成？

My colleague suggested a busy loop within the parent process that checks if the child process has finished. Surely there is a better way.

我的同事建议在父进程中使用一个忙循环来检查子进程是否已完成。当然还有更好的方法。

In Unix, when a process is forked, a signal handler is called from within the parent process when the child process has finished. But I cannot see anything like that in Python. Am I missing something?

在 Unix 中，当进程被 fork 时，当子进程完成时，从父进程内部调用信号处理程序。但是我在 Python 中看不到类似的东西。我错过了什么吗？

How is it that the end of a child process can be observed from within the parent process? (Of course, I do not want to call Process.join() as it would freeze up the GUI interface.)

如何从父进程中观察到子进程的结束？（当然，我不想调用 Process.join() 因为它会冻结 GUI 界面。）

This question is not limited to multi-processing: I have exactly the same problem with multi-threading.

这个问题不仅限于多处理：我对多线程有完全相同的问题。

采纳答案by Matthew Walker

This answer is really simple! (It just took me daysto work it out.)

这个答案真的很简单！（我花了几天时间才弄明白。）

Combined with PyGTK's idle_add(), you can create an AutoJoiningThread. The total code is borderline trivial:

结合 PyGTK 的 idle_add()，你可以创建一个 AutoJoiningThread。总代码是微不足道的：

class AutoJoiningThread(threading.Thread):
    def run(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)

If you want to do more than just join (such as collecting results) then you can extend the above class to emit signals on completion, as is done in the following example:

如果您想做的不仅仅是加入（例如收集结果），那么您可以扩展上述类以在完成时发出信号，如下例所示：

import threading
import time
import sys
import gobject
gobject.threads_init()

class Child:
    def __init__(self):
        self.result = None

    def play(self, count):
        print "Child starting to play."
        for i in range(count):
            print "Child playing."
            time.sleep(1)
        print "Child finished playing."
        self.result = 42

    def get_result(self, obj):
        print "The result was "+str(self.result)

class AutoJoiningThread(threading.Thread, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
        }

    def __init__(self, *args, **kwargs):
        threading.Thread.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    def run(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)
        gobject.idle_add(self.emit, 'finished')

    def join(self):
        threading.Thread.join(self)
        print "Called Thread.join()"

if __name__ == '__main__':
    print "Creating child"
    child = Child()
    print "Creating thread"
    thread = AutoJoiningThread(target=child.play,
                               args=(3,))
    thread.connect('finished', child.get_result)
    print "Starting thread"
    thread.start()
    print "Running mainloop (Ctrl+C to exit)"
    mainloop = gobject.MainLoop()

    try:
        mainloop.run()
    except KeyboardInterrupt:
        print "Received KeyboardInterrupt.  Quiting."
        sys.exit()

    print "God knows how we got here.  Quiting."
    sys.exit()

The output of the above example will depend on the order the threads are executed, but it will be similar to:

上述示例的输出将取决于线程的执行顺序，但类似于：

Creating child
Creating thread
Starting thread
Child starting to play.
 Child playing.
Running mainloop (Ctrl+C to exit)
Child playing.
Child playing.
Child finished playing.
Called Thread.join()
The result was 42
^CReceived KeyboardInterrupt.  Quiting.

It's not possible to create an AutoJoiningProcess in the same way (because we cannot call idle_add() across two different processes), however we can use an AutoJoiningThread to get what we want:

不可能以相同的方式创建 AutoJoiningProcess（因为我们不能跨两个不同的进程调用 idle_add()），但是我们可以使用 AutoJoiningThread 来获得我们想要的：

class AutoJoiningProcess(multiprocessing.Process):
    def start(self):
        thread = AutoJoiningThread(target=self.start_process)
        thread.start() # automatically joins

    def start_process(self):
        multiprocessing.Process.start(self)
        self.join()

To demonstrate AutoJoiningProcess here is another example:

为了演示 AutoJoiningProcess 这里是另一个例子：

import threading
import multiprocessing
import time
import sys
import gobject
gobject.threads_init()

class Child:
    def __init__(self):
        self.result = multiprocessing.Manager().list()

    def play(self, count):
        print "Child starting to play."
        for i in range(count):
            print "Child playing."
            time.sleep(1)
    print "Child finished playing."
        self.result.append(42)

    def get_result(self, obj):
        print "The result was "+str(self.result)

class AutoJoiningThread(threading.Thread, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
    }

    def __init__(self, *args, **kwargs):
        threading.Thread.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    def run(self):
        threading.Thread.run(self)
        gobject.idle_add(self.join)
        gobject.idle_add(self.emit, 'finished')

    def join(self):
        threading.Thread.join(self)
        print "Called Thread.join()"

class AutoJoiningProcess(multiprocessing.Process, gobject.GObject):
    __gsignals__ = {
        'finished': (gobject.SIGNAL_RUN_LAST,
                     gobject.TYPE_NONE,
                     ())
        }

    def __init__(self, *args, **kwargs):
        multiprocessing.Process.__init__(self, *args, **kwargs)
        gobject.GObject.__init__(self)

    def start(self):
        thread = AutoJoiningThread(target=self.start_process)
        thread.start()

    def start_process(self):
        multiprocessing.Process.start(self)
        self.join()
        gobject.idle_add(self.emit, 'finished')

    def join(self):
        multiprocessing.Process.join(self)
        print "Called Process.join()"

if __name__ == '__main__':
    print "Creating child"
    child = Child()
    print "Creating thread"
    process = AutoJoiningProcess(target=child.play,
                               args=(3,))
    process.connect('finished',child.get_result)
    print "Starting thread"
    process.start()
    print "Running mainloop (Ctrl+C to exit)"
    mainloop = gobject.MainLoop()

    try:
        mainloop.run()
    except KeyboardInterrupt:
        print "Received KeyboardInterrupt.  Quiting."
        sys.exit()

    print "God knows how we got here.  Quiting."
    sys.exit()

The resulting output will be very similar to the example above, except this time we have both the process joining and it's attendant thread joining too:

结果输出将与上面的示例非常相似，除了这次我们同时加入了进程和它的伴随线程：

Creating child
Creating thread
Starting thread
Running mainloop (Ctrl+C to exit)
 Child starting to play.
Child playing.
Child playing.
Child playing.
Child finished playing.
Called Process.join()
The result was [42]
Called Thread.join()
^CReceived KeyboardInterrupt.  Quiting.

Unfortunately:

很遗憾：

1. This solution is dependent on gobject, due to the use of idle_add(). gobject is used by PyGTK.
2. This is not a true parent/child relationship. If one of these threads is started by another thread, then it will nonetheless be joined by the thread running the mainloop, not the parent thread. This problem holds true for AutoJoiningProcess too, except there I imagine an exception would be thrown.

1. 由于使用了 idle_add()，此解决方案依赖于 gobject。PyGTK 使用 gobject。
2. 这不是真正的父子关系。如果这些线程中的一个由另一个线程启动，那么它仍然会被运行主循环的线程加入，而不是父线程。这个问题也适用于 AutoJoiningProcess，除非我认为会抛出异常。

Thus to use this approach, it would be best to only create threads/process from within the mainloop/GUI.

因此，要使用这种方法，最好仅从主循环/GUI 内创建线程/进程。

回答by nmichaels

You can use a queueto communicate with child processes. You can stick intermediate results on it, or messages indicating that milestones have been hit (for progress bars) or just a message indicating that the process is ready to be joined. Polling it with emptyis easy and fast.

您可以使用队列与子进程通信。您可以在其上粘贴中间结果，或表明已达到里程碑的消息（对于进度条）或仅表明已准备好加入流程的消息。用空轮询它既简单又快速。

If you really only want to know if it's done, you can watch the exitcodeof your process or poll is_alive().

如果你真的只是想知道，如果它这样做，你可以观赏到退出码的过程中或投票表决is_alive（） 。

回答by Andy Skirrow

have a look at the subprocess module:

看看子流程模块：

http://docs.python.org/library/subprocess.html

http://docs.python.org/library/subprocess.html

import subprocess
let pipe = subprocess.Popen("ls -l", stdout=subprocess.PIPE)
allText = pipe.stdout.read()
pipe.wait()
retVal = pipe.returncode

回答by Matthew Walker

In my efforts to try to find an answer to my own question, I stumbled across PyGTK's idle_add() function. This gives me the following possibility:

在我努力寻找自己问题的答案时，我偶然发现了 PyGTK 的idle_add() 函数。这给了我以下可能性：

1. Create a new child process that communicates via a Queue.
2. Create a listener thread that listens to the Queue, when the child process sends the listener a message saying that it is finished, the listener calls idle_add() that sets up a callback.
3. During the next time around the main loop the parent process will call the callback.
4. The callback can extract results, join the child process and join the listener-thread.

1. 创建一个通过队列进行通信的新子进程。
2. 创建一个侦听队列的侦听器线程，当子进程向侦听器发送消息说它已完成时，侦听器调用 idle_add() 设置回调。
3. 在下一次主循环中，父进程将调用回调。
4. 回调可以提取结果，加入子进程，加入监听线程。

This seems an overly complex way to re-create Unix's call-callback-when-child-process-is-done.

这似乎是一种过于复杂的方法来重新创建 Unix 的 call-callback-when-child-process-is-done。

This must be an uber-common problem with GUIs in Python. Surely there is a standard pattern to solve this problem?

这一定是 Python 中 GUI 的一个非常普遍的问题。当然有解决这个问题的标准模式吗？

回答by manifest

I think as a part of making python multi-platform, simple things like SIGCHLD must be done yourself. Agreed, this is a little more work when all you want to do is know when the child is done, but it really isn't THAT painful. Consider the following that uses a child process to do the work, two multiprocessing.Event instances, and a thread to check if the child process is done:

我认为作为使python多平台的一部分，像SIGCHLD这样的简单事情必须自己完成。同意，当您只想知道孩子什么时候完成时，这需要做更多的工作，但实际上并没有那么痛苦。考虑以下使用子进程来完成工作、两个 multiprocessing.Event 实例和一个线程来检查子进程是否完成的情况：

import threading
from multiprocessing import Process, Event
from time import sleep

def childsPlay(event):
    print "Child started"
    for i in range(3):
        print "Child is playing..."
        sleep(1)
    print "Child done"
    event.set()

def checkChild(event, killEvent):
    event.wait()
    print "Child checked, and is done playing"
    if raw_input("Do again? y/n:") == "y":
        event.clear()
        t = threading.Thread(target=checkChild, args=(event, killEvent))
        t.start()
        p = Process(target=childsPlay, args=(event,))
        p.start()
    else:
        cleanChild()
        killEvent.set()

def cleanChild():
    print "Cleaning up the child..."

if __name__ == '__main__':
    event = Event()
    killEvent = Event()

    # process to do work
    p = Process(target=childsPlay, args=(event,))
    p.start()

    # thread to check on child process
    t = threading.Thread(target=checkChild, args=(event, killEvent))
    t.start()

    try:
        while not killEvent.is_set():
            print "GUI running..."
            sleep(1)
    except KeyboardInterrupt:
        print "Quitting..."
        exit(0)
    finally:
        print "Main done"

EDIT

编辑

Joining to all processes and threads created is a good practice because it will help indicate when zombie (never-finishing) processes/threads are being created. I've altered the above code making a ChildChecker class that inherits from threading.Thread. It's sole purpose is to start a job in a separate process, wait for that process to finish, and then notify the GUI when everything is complete. Joining on the ChildChecker will also join the process it is "checking". Now, if the process doesn't join after 5 seconds, the thread will force terminate the process. Enter "y" creates starts a child process running "endlessChildsPlay" that must demonstrate force termination.

加入创建的所有进程和线程是一个很好的做法，因为它有助于指示何时创建僵尸（永不完成）进程/线程。我修改了上面的代码，创建了一个继承自 threading.Thread 的 ChildChecker 类。它的唯一目的是在单独的进程中启动一个作业，等待该进程完成，然后在一切完成时通知 GUI。加入 ChildChecker 也将加入它正在“检查”的过程。现在，如果进程在 5 秒后没有加入，线程将强制终止进程。输入“y”将创建一个运行“endlessChildsPlay”的子进程，该进程必须证明强制终止。

import threading
from multiprocessing import Process, Event
from time import sleep

def childsPlay(event):
    print "Child started"
    for i in range(3):
        print "Child is playing..."
        sleep(1)
    print "Child done"
    event.set()

def endlessChildsPlay(event):
    print "Endless child started"
    while True:
        print "Endless child is playing..."
        sleep(1)
        event.set()
    print "Endless child done"

class ChildChecker(threading.Thread):
    def __init__(self, killEvent):
        super(ChildChecker, self).__init__()
        self.killEvent = killEvent
        self.event = Event()
        self.process = Process(target=childsPlay, args=(self.event,))

    def run(self):
        self.process.start()

        while not self.killEvent.is_set():
            self.event.wait()
            print "Child checked, and is done playing"
            if raw_input("Do again? y/n:") == "y":
                self.event.clear()
                self.process = Process(target=endlessChildsPlay, args=(self.event,))
                self.process.start()
            else:
                self.cleanChild()
                self.killEvent.set()

    def join(self):
        print "Joining child process"
        # Timeout on 5 seconds
        self.process.join(5)

        if self.process.is_alive():
            print "Child did not join!  Killing.."
            self.process.terminate()
        print "Joining ChildChecker thread"
        super(ChildChecker, self).join()


    def cleanChild(self):
        print "Cleaning up the child..."

if __name__ == '__main__':
    killEvent = Event()
    # thread to check on child process
    t = ChildChecker(killEvent)
    t.start()

    try:
        while not killEvent.is_set():
            print "GUI running..."
            sleep(1)
    except KeyboardInterrupt:
        print "Quitting..."
        exit(0)
    finally:
        t.join()
        print "Main done"