Python 线程和多处理模块之间有什么区别？

What are the differences between the threading and multiprocessing modules?

提问by lucacerone

I am learning how to use the threadingand the multiprocessingmodules in Python to run certain operations in parallel and speed up my code.

我正在学习如何使用Python 中的threading和multiprocessing模块来并行运行某些操作并加速我的代码。

I am finding this hard (maybe because I don't have any theoretical background about it) to understand what the difference is between a threading.Thread()object and a multiprocessing.Process()one.

我发现这很难（可能是因为我没有任何理论背景）来理解threading.Thread()物体和物体之间的区别multiprocessing.Process()。

Also, it is not entirely clear to me how to instantiate a queue of jobs and having only 4 (for example) of them running in parallel, while the other wait for resources to free before being executed.

此外，我并不完全清楚如何实例化一个作业队列并且只有 4 个（例如）并行运行，而另一个在执行之前等待资源释放。

I find the examples in the documentation clear, but not very exhaustive; as soon as I try to complicate things a bit, I receive a lot of weird errors (like a method that can't be pickled, and so on).

我发现文档中的例子很清楚，但不是很详尽；一旦我尝试将事情复杂化，我就会收到很多奇怪的错误（例如无法腌制的方法等）。

So, when should I use the threadingand multiprocessingmodules?

那么，我应该什么时候使用threading和multiprocessing模块？

Can you link me to some resources that explain the concepts behind these two modules and how to use them properly for complex tasks?

您能否将我链接到一些解释这两个模块背后的概念以及如何正确使用它们来完成复杂任务的资源？

采纳答案by abarnert

What Giulio Franco saysis true for multithreading vs. multiprocessing in general.

Giulio Franco 所说的对于多线程与一般多处理是正确的。

However, Python^*has an added issue: There's a Global Interpreter Lock that prevents two threads in the same process from running Python code at the same time. This means that if you have 8 cores, and change your code to use 8 threads, it won't be able to use 800% CPU and run 8x faster; it'll use the same 100% CPU and run at the same speed. (In reality, it'll run a little slower, because there's extra overhead from threading, even if you don't have any shared data, but ignore that for now.)

但是，Python ^*有一个附加问题：有一个全局解释器锁，可以防止同一进程中的两个线程同时运行 Python 代码。这意味着如果您有 8 个内核，并且将您的代码更改为使用 8 个线程，它将无法使用 800% 的 CPU 并且运行速度提高 8 倍；它将使用相同的 100% CPU 并以相同的速度运行。（实际上，它的运行速度会慢一点，因为线程会产生额外的开销，即使您没有任何共享数据，但现在先忽略它。）

There are exceptions to this. If your code's heavy computation doesn't actually happen in Python, but in some library with custom C code that does proper GIL handling, like a numpy app, you will get the expected performance benefit from threading. The same is true if the heavy computation is done by some subprocess that you run and wait on.

也有例外。如果您的代码的繁重计算实际上不是在 Python 中发生，而是在一些带有自定义 C 代码的库中进行适当的 GIL 处理，例如 numpy 应用程序，您将从线程中获得预期的性能优势。如果繁重的计算是由您运行并等待的某个子进程完成的，情况也是如此。

More importantly, there are cases where this doesn't matter. For example, a network server spends most of its time reading packets off the network, and a GUI app spends most of its time waiting for user events. One reason to use threads in a network server or GUI app is to allow you to do long-running "background tasks" without stopping the main thread from continuing to service network packets or GUI events. And that works just fine with Python threads. (In technical terms, this means Python threads give you concurrency, even though they don't give you core-parallelism.)

更重要的是，有些情况下这无关紧要。例如，网络服务器花费大部分时间从网络读取数据包，而 GUI 应用程序花费大部分时间等待用户事件。在网络服务器或 GUI 应用程序中使用线程的原因之一是允许您执行长时间运行的“后台任务”，而无需停止主线程继续为网络数据包或 GUI 事件提供服务。这对于 Python 线程来说工作得很好。（从技术角度来说，这意味着 Python 线程可以提供并发性，即使它们不提供核心并行性。）

But if you're writing a CPU-bound program in pure Python, using more threads is generally not helpful.

但是，如果您正在用纯 Python 编写 CPU 密集型程序，则使用更多线程通常没有帮助。

Using separate processes has no such problems with the GIL, because each process has its own separate GIL. Of course you still have all the same tradeoffs between threads and processes as in any other languages—it's more difficult and more expensive to share data between processes than between threads, it can be costly to run a huge number of processes or to create and destroy them frequently, etc. But the GIL weighs heavily on the balance toward processes, in a way that isn't true for, say, C or Java. So, you will find yourself using multiprocessing a lot more often in Python than you would in C or Java.

使用单独的进程对 GIL 没有这样的问题，因为每个进程都有自己单独的 GIL。当然，线程和进程之间的权衡仍然与任何其他语言相同——在进程之间共享数据比在线程之间共享数据更困难、更昂贵，运行大量进程或创建和销毁的成本可能很高它们经常出现，等等。但是 GIL 对进程的平衡有很大的影响，这种方式对于 C 或 Java 来说并非如此。因此，您会发现自己在 Python 中比在 C 或 Java 中更频繁地使用多处理。

---

---

Meanwhile, Python's "batteries included" philosophy brings some good news: It's very easy to write code that can be switched back and forth between threads and processes with a one-liner change.

同时，Python 的“包含电池”的理念带来了一些好消息：编写代码非常容易，只需进行一次更改即可在线程和进程之间来回切换。

If you design your code in terms of self-contained "jobs" that don't share anything with other jobs (or the main program) except input and output, you can use the concurrent.futureslibrary to write your code around a thread pool like this:

如果您根据自包含的“作业”设计代码，除了输入和输出外，不与其他作业（或主程序）共享任何内容，您可以使用该concurrent.futures库围绕线程池编写代码，如下所示：

with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
    executor.submit(job, argument)
    executor.map(some_function, collection_of_independent_things)
    # ...

You can even get the results of those jobs and pass them on to further jobs, wait for things in order of execution or in order of completion, etc.; read the section on Futureobjects for details.

您甚至可以获取这些作业的结果并将它们传递给进一步的作业，按执行顺序或完成顺序等待事物等；Future有关详细信息，请阅读有关对象的部分。

Now, if it turns out that your program is constantly using 100% CPU, and adding more threads just makes it slower, then you're running into the GIL problem, so you need to switch to processes. All you have to do is change that first line:

现在，如果事实证明您的程序一直在使用 100% CPU，并且添加更多线程只会使其变慢，那么您就会遇到 GIL 问题，因此您需要切换到进程。您所要做的就是更改第一行：

with concurrent.futures.ProcessPoolExecutor(max_workers=4) as executor:

The only real caveat is that your jobs' arguments and return values have to be pickleable (and not take too much time or memory to pickle) to be usable cross-process. Usually this isn't a problem, but sometimes it is.

唯一真正需要注意的是，您的作业的参数和返回值必须是可腌制的（并且不会花费太多时间或内存来腌制）才能跨进程使用。通常这不是问题，但有时是。

---

---

But what if your jobs can't be self-contained? If you can design your code in terms of jobs that pass messagesfrom one to another, it's still pretty easy. You may have to use threading.Threador multiprocessing.Processinstead of relying on pools. And you will have to create queue.Queueor multiprocessing.Queueobjects explicitly. (There are plenty of other options—pipes, sockets, files with flocks, … but the point is, you have to do somethingmanually if the automatic magic of an Executor is insufficient.)

但是，如果您的工作不能自给自足怎么办？如果您可以根据将消息从一个传递到另一个的作业来设计您的代码，那仍然很容易。您可能必须使用threading.Thread或multiprocessing.Process代替依赖池。并且您必须显式地创建queue.Queue或创建multiprocessing.Queue对象。（还有很多其他选项——管道、套接字、文件群……但重点是，如果 Executor 的自动魔法不足，你必须手动做一些事情。）

But what if you can't even rely on message passing? What if you need two jobs to both mutate the same structure, and see each others' changes? In that case, you will need to do manual synchronization (locks, semaphores, conditions, etc.) and, if you want to use processes, explicit shared-memory objects to boot. This is when multithreading (or multiprocessing) gets difficult. If you can avoid it, great; if you can't, you will need to read more than someone can put into an SO answer.

但是，如果您甚至不能依靠消息传递呢？如果你需要两个工作来改变相同的结构，并看到彼此的变化怎么办？在这种情况下，您将需要进行手动同步（锁、信号量、条件等），并且如果您想使用进程，则需要显式共享内存对象来引导。这是多线程（或多处理）变得困难的时候。如果你能避免它，那太好了；如果您不能，您将需要阅读的内容比某人可以放入 SO 答案中的多。

---

---

From a comment, you wanted to know what's different between threads and processes in Python. Really, if you read Giulio Franco's answer and mine and all of our links, that should cover everything…?but a summary would definitely be useful, so here goes:

从评论中，您想知道 Python 中的线程和进程之间有什么不同。真的，如果你阅读 Giulio Franco 的回答和我的以及我们所有的链接，那应该涵盖所有内容......但总结肯定是有用的，所以这里是：

1. Threads share data by default; processes do not.
2. As a consequence of (1), sending data between processes generally requires pickling and unpickling it.^**
3. As another consequence of (1), directly sharing data between processes generally requires putting it into low-level formats like Value, Array, and ctypestypes.
4. Processes are not subject to the GIL.
5. On some platforms?(mainly Windows), processes are much more expensive to create and destroy.
6. There are some extra restrictions on processes, some of which are different on different platforms. See Programming guidelinesfor details.
7. The threadingmodule doesn't have some of the features of the multiprocessingmodule. (You can use multiprocessing.dummyto get most of the missing API on top of threads, or you can use higher-level modules like concurrent.futuresand not worry about it.)

1. 线程默认共享数据；进程没有。
2. 作为 (1) 的结果，在进程之间发送数据通常需要对其进行酸洗和取消酸洗。^**
3. 作为 (1) 的另一个结果，在进程之间直接共享数据通常需要将其放入低级格式，如值、数组和ctypes类型。
4. 流程不受 GIL 约束。
5. 在某些平台上？（主要是 Windows），创建和销毁进程的成本要高得多。
6. 对进程有一些额外的限制，其中一些在不同平台上是不同的。有关详细信息，请参阅编程指南。
7. 该threading模块不具有该模块的某些功能multiprocessing。（您可以使用multiprocessing.dummy在线程之上获取大部分缺失的 API，或者您可以使用更高级别的模块，例如concurrent.futures而不必担心。）

---

---

_{* It's not actually Python, the language, that has this issue, but CPython, the "standard" implementation of that language. Some other implementations don't have a GIL, like Jython.}

_{* 实际上，存在此问题的不是 Python，即语言，而是 CPython，即该语言的“标准”实现。其他一些实现没有 GIL，例如 Jython。}

_{** If you're using the forkstart method for multiprocessing—which you can on most non-Windows platforms—each child process gets any resources the parent had when the child was started, which can be another way to pass data to children.}

_{** 如果您使用forkstart 方法进行多处理——您可以在大多数非 Windows 平台上使用这种方法——每个子进程在子进程启动时获取父进程拥有的任何资源，这可能是将数据传递给子进程的另一种方式。}

回答by Giulio Franco

Multiple threads can exist in a single process. The threads that belong to the same process share the same memory area (can read from and write to the very same variables, and can interfere with one another). On the contrary, different processes live in different memory areas, and each of them has its own variables. In order to communicate, processes have to use other channels (files, pipes or sockets).

单个进程中可以存在多个线程。属于同一进程的线程共享相同的内存区域（可以读取和写入完全相同的变量，并且可以相互干扰）。相反，不同的进程生活在不同的内存区域，每个进程都有自己的变量。为了进行通信，进程必须使用其他通道（文件、管道或套接字）。

If you want to parallelize a computation, you're probably going to need multithreading, because you probably want the threads to cooperate on the same memory.

如果要并行化计算，则可能需要多线程，因为您可能希望线程在同一内存上协作。

Speaking about performance, threads are faster to create and manage than processes (because the OS doesn't need to allocate a whole new virtual memory area), and inter-thread communication is usually faster than inter-process communication. But threads are harder to program. Threads can interfere with one another, and can write to each other's memory, but the way this happens is not always obvious (due to several factors, mainly instruction reordering and memory caching), and so you are going to need synchronization primitives to control access to your variables.

说到性能，线程的创建和管理速度比进程快（因为操作系统不需要分配一个全新的虚拟内存区域），并且线程间通信通常比进程间通信快。但是线程更难编程。线程可以相互干扰，并且可以写入彼此的内存，但是这种情况发生的方式并不总是很明显（由于多种因素，主要是指令重新排序和内存缓存），因此您将需要同步原语来控制访问到你的变量。

回答by innosam

Well, most of the question is answered by Giulio Franco. I will further elaborate on the consumer-producer problem, which I suppose will put you on the right track for your solution to using a multithreaded app.

好吧，Giulio Franco 回答了大部分问题。我将进一步详细说明消费者 - 生产者问题，我想这将使您走上正确的轨道，以解决使用多线程应用程序的问题。

fill_count = Semaphore(0) # items produced
empty_count = Semaphore(BUFFER_SIZE) # remaining space
buffer = Buffer()

def producer(fill_count, empty_count, buffer):
    while True:
        item = produceItem()
        empty_count.down();
        buffer.push(item)
        fill_count.up()

def consumer(fill_count, empty_count, buffer):
    while True:
        fill_count.down()
        item = buffer.pop()
        empty_count.up()
        consume_item(item)

You could read more on the synchronization primitives from:

您可以从以下位置阅读有关同步原语的更多信息：

 http://linux.die.net/man/7/sem_overview
 http://docs.python.org/2/library/threading.html

The pseudocode is above. I suppose you should search the producer-consumer-problem to get more references.

伪代码如上。我想你应该搜索生产者消费者问题以获得更多参考。

回答by ehfaafzv

I believe this linkanswers your question in an elegant way.

我相信此链接以优雅的方式回答了您的问题。

To be short, if one of your sub-problems has to wait while another finishes, multithreading is good (in I/O heavy operations, for example); by contrast, if your sub-problems could really happen at the same time, multiprocessing is suggested. However, you won't create more processes than your number of cores.

简而言之，如果你的一个子问题必须等待另一个完成，多线程是好的（例如在 I/O 繁重的操作中）；相比之下，如果您的子问题真的可以同时发生，则建议进行多处理。但是，您创建的进程不会超过内核数。

回答by Mario Aguilera

Here's some performance data for python 2.6.x that calls to question the notion that threading is more performant that multiprocessing in IO-bound scenarios. These results are from a 40-processor IBM System x3650 M4 BD.

这是 python 2.6.x 的一些性能数据，它要求质疑线程在 IO 绑定场景中比多处理性能更高的概念。这些结果来自 40 个处理器的 IBM System x3650 M4 BD。

IO-Bound Processing : Process Pool performed better than Thread Pool

IO-Bound 处理：进程池比线程池表现更好

>>> do_work(50, 300, 'thread','fileio')
do_work function took 455.752 ms

>>> do_work(50, 300, 'process','fileio')
do_work function took 319.279 ms

CPU-Bound Processing : Process Pool performed better than Thread Pool

CPU-Bound Processing：进程池比线程池表现更好

>>> do_work(50, 2000, 'thread','square')
do_work function took 338.309 ms

>>> do_work(50, 2000, 'process','square')
do_work function took 287.488 ms

These aren't rigorous tests, but they tell me that multiprocessing isn't entirely unperformant in comparison to threading.

这些不是严格的测试，但它们告诉我，与线程相比，多处理并非完全没有性能。

Code used in the interactive python console for the above tests

用于上述测试的交互式 python 控制台中的代码

from multiprocessing import Pool
from multiprocessing.pool import ThreadPool
import time
import sys
import os
from glob import glob

text_for_test = str(range(1,100000))

def fileio(i):
 try :
  os.remove(glob('./test/test-*'))
 except : 
  pass
 f=open('./test/test-'+str(i),'a')
 f.write(text_for_test)
 f.close()
 f=open('./test/test-'+str(i),'r')
 text = f.read()
 f.close()


def square(i):
 return i*i

def timing(f):
 def wrap(*args):
  time1 = time.time()
  ret = f(*args)
  time2 = time.time()
  print '%s function took %0.3f ms' % (f.func_name, (time2-time1)*1000.0)
  return ret
 return wrap

result = None

@timing
def do_work(process_count, items, process_type, method) :
 pool = None
 if process_type == 'process' :
  pool = Pool(processes=process_count)
 else :
  pool = ThreadPool(processes=process_count)
 if method == 'square' : 
  multiple_results = [pool.apply_async(square,(a,)) for a in range(1,items)]
  result = [res.get()  for res in multiple_results]
 else :
  multiple_results = [pool.apply_async(fileio,(a,)) for a in range(1,items)]
  result = [res.get()  for res in multiple_results]


do_work(50, 300, 'thread','fileio')
do_work(50, 300, 'process','fileio')

do_work(50, 2000, 'thread','square')
do_work(50, 2000, 'process','square')