Python 3.4 中的多处理 vs 多线程 vs asyncio

multiprocessing vs multithreading vs asyncio in Python 3.4

提问by user3654650

I found that in Python 3.4 there are few different libraries for multiprocessing/threading: multiprocessingvs threadingvs asyncio.

我发现在 Python 3.4 中，用于多处理/线程的库很少：multiprocessingvs threadingvs asyncio。

But I don't know which one to use or is the "recommended one". Do they do the same thing, or are different? If so, which one is used for what? I want to write a program that uses multicores in my computer. But I don't know which library I should learn.

但我不知道该使用哪一个，或者是“推荐的”。他们做同样的事情，还是不同？如果是的话，哪个是用来做什么的？我想编写一个在我的计算机中使用多核的程序。但是我不知道我应该学习哪个库。

采纳答案by user3159253

They are intended for (slightly) different purposes and/or requirements. CPython (a typical, mainline Python implementation) still has the global interpreter lockso a multi-threaded application (a standard way to implement parallel processing nowadays) is suboptimal. That's why multiprocessingmaybe preferred over threading. But not every problem may be effectively split into [almost independent] pieces, so there may be a need in heavy interprocess communications. That's why multiprocessingmay not be preferred over threadingin general.

它们用于（略微）不同的目的和/或要求。CPython（典型的主流 Python 实现）仍然具有全局解释器锁，因此多线程应用程序（当今实现并行处理的标准方法）不是最佳选择。这就是为什么multiprocessing可能优于threading. 但并非每个问题都可以有效地拆分为[几乎独立的] 部分，因此可能需要大量进程间通信。这就是为什么在一般情况下multiprocessing可能不受欢迎的原因threading。

asyncio(this technique is available not only in Python, other languages and/or frameworks also have it, e.g. Boost.ASIO) is a method to effectively handle a lot of I/O operations from many simultaneous sources w/o need of parallel code execution. So it's just a solution (a good one indeed!) for a particular task, not for parallel processing in general.

asyncio（这种技术不仅在 Python 中可用，其他语言和/或框架也有，例如Boost.ASIO）是一种有效处理来自许多同时源的大量 I/O 操作的方法，无需并行代码执行. 所以它只是针对特定任务的解决方案（确实是一个很好的解决方案！），而不是一般的并行处理。

回答by Benyamin Jafari

[Quick Answer]

[快速回答]

TL;DR

TL; 博士

Making the Right Choice:

做出正确的选择：

We have walked through the most popular forms of concurrency. But the question remains - when should choose which one? It really depends on the use cases. From my experience (and reading), I tend to follow this pseudo code:

我们已经了解了最流行的并发形式。但问题仍然存在——什么时候应该选择哪一个？这真的取决于用例。根据我的经验（和阅读），我倾向于遵循以下伪代码：

if io_bound:
    if io_very_slow:
        print("Use Asyncio")
    else:
        print("Use Threads")
else:
    print("Multi Processing")

• CPU Bound => Multi Processing
• I/O Bound, Fast I/O, Limited Number of Connections => Multi Threading
• I/O Bound, Slow I/O, Many connections => Asyncio

• CPU 绑定 => 多处理
• I/O 绑定、快速 I/O、有限连接数 => 多线程
• I/O 绑定、慢速 I/O、许多连接 => Asyncio

Reference

参考

---

---

[NOTE]:

[注意]：

• If you have a long call method (i.e. a method that contained with a sleep time or lazy I/O), the best choice is asyncio, Twistedor Tornadoapproach (coroutine methods), that works with a single thread as concurrency.
• asyncioworks on Python3.4and later.
• Tornadoand Twistedare ready since Python2.7
• uvloopis ultra fast asyncioevent loop (uvloopmakes asyncio2-4x faster).

• 如果您有一个长调用方法（即包含睡眠时间或惰性 I/O 的方法），最好的选择是asyncio、Twisted或Tornado方法（协程方法），它使用单线程作为并发。
• asyncio适用于Python3.4及更高版本。
• Tornado和Twisted从Python2.7开始准备就绪
• uvloop是超快速asyncio事件循环（uvloop使asyncio速度提高 2-4 倍）。

---

---

[UPDATE (2019)]:

[更新（2019）]：

• Japranto ^(GitHub)is a very fast pipelining HTTP server based on uvloop.

• Japranto ^{( GitHub)}是一个基于uvloop的非常快速的流水线 HTTP 服务器。

回答by Farsheed

This is the basic idea:

这是基本思想：

Is it IO-BOUND ? ---------> USE asyncio

IS IT CPU-HEAVY ? -----> USE multiprocessing

ELSE ? ----------------------> USE threading

是IO绑定吗？---------> 使用asyncio

它是CPU-HEAVY 吗？-----> 使用multiprocessing

别的 ？--------------> 使用threading

So basically stick to threading unless you have IO/CPU problems.

所以基本上坚持线程，除非你有 IO/CPU 问题。