Python 多处理模块中的 ThreadPool 与 Pool 有什么区别？

What's the difference between ThreadPool vs Pool in the multiprocessing module?

提问by ozn

Whats the difference between ThreadPooland Poolin multiprocessingmodule. When I try my code out, this is the main difference I see:

ThreadPool和Pool在multiprocessing模块中有什么区别。当我尝试我的代码时，这是我看到的主要区别：

from multiprocessing import Pool
import os, time

print("hi outside of main()")

def hello(x):
    print("inside hello()")
    print("Proccess id: ", os.getpid())
    time.sleep(3)
    return x*x

if __name__ == "__main__":
    p = Pool(5)
    pool_output = p.map(hello, range(3))

    print(pool_output)

I see the following output:

我看到以下输出：

hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
hi outside of main()
inside hello()
Proccess id:  13268
inside hello()
Proccess id:  11104
inside hello()
Proccess id:  13064
[0, 1, 4]

With "ThreadPool":

使用“线程池”：

from multiprocessing.pool import ThreadPool
import os, time

print("hi outside of main()")

def hello(x):
    print("inside hello()")
    print("Proccess id: ", os.getpid())
    time.sleep(3)
    return x*x

if __name__ == "__main__":
    p = ThreadPool(5)
    pool_output = p.map(hello, range(3))

    print(pool_output)

I see the following output:

我看到以下输出：

hi outside of main()
inside hello()
inside hello()
Proccess id:  15204
Proccess id:  15204
inside hello()
Proccess id:  15204
[0, 1, 4]

My questions are:

我的问题是：

• why is the “outside __main__()” run each time in the Pool?

• multiprocessing.pool.ThreadPooldoesn't spawn new processes? It just creates new threads?

• If so whats the difference between using multiprocessing.pool.ThreadPoolas opposed to just threadingmodule?

• 为什么每次都在 __main__() 之外运行Pool？

• multiprocessing.pool.ThreadPool不会产生新的进程？它只是创建新线程？

• 如果是这样，使用multiprocessing.pool.ThreadPool与仅使用threading模块有什么区别？

I don't see any official documentation for ThreadPoolanywhere, can someone help me out where I can find it?

我在任何ThreadPool地方都没有看到任何官方文档，有人可以帮我找到它吗？

回答by noxdafox

The multiprocessing.pool.ThreadPoolbehaves the same as the multiprocessing.Poolwith the only difference that uses threads instead of processes to run the workers logic.

该multiprocessing.pool.ThreadPool行为一样multiprocessing.Pool，唯一的区别在于使用线程，而不是进程运行的工人逻辑。

The reason you see

你看到的原因

hi outside of main()

being printed multiple times with the multiprocessing.Poolis due to the fact that the pool will spawn5 independent processes. Each process will initialize its own Python interpreter and load the module resulting in the top level printbeing executed again.

被多次打印multiprocessing.Pool是因为池将产生5 个独立进程。每个进程将初始化自己的 Python 解释器并加载模块，从而print再次执行顶层。

Note that this happens only if the spawnprocess creation method is used (only method available on Windows). If you use the forkone (Unix), you will see the message printed only once as for the threads.

请注意，只有在使用spawn进程创建方法时才会发生这种情况（仅在 Windows 上可用的方法）。如果您使用fork一个（Unix），您将看到仅打印一次线程的消息。

The multiprocessing.pool.ThreadPoolis not documented as its implementation has never been completed. It lacks tests and documentation. You can see its implementation in the source code.

将multiprocessing.pool.ThreadPool作为其实现从未完成未记录。它缺乏测试和文档。您可以在源代码中看到它的实现。

I believe the next natural question is: when to use a thread based pool and when to use a process based one?

我相信下一个自然的问题是：何时使用基于线程的池以及何时使用基于进程的池？

The rule of thumb is:

经验法则是：

• IO bound jobs -> multiprocessing.pool.ThreadPool
• CPU bound jobs -> multiprocessing.Pool
• Hybrid jobs -> depends on the workload, I usually prefer the multiprocessing.Pooldue to the advantage process isolation brings

• IO 绑定作业 -> multiprocessing.pool.ThreadPool
• CPU 绑定作业 -> multiprocessing.Pool
• 混合作业 -> 取决于工作量，multiprocessing.Pool由于进程隔离带来的优势，我通常更喜欢

On Python 3 you might want to take a look at the concurrent.future.Executorpool implementations.

在 Python 3 上，您可能需要查看concurrent.future.Executor池实现。