使用 Python 多处理的高内存使用率

High Memory Usage Using Python Multiprocessing

提问by Goutham

I have seen a couple of posts on memory usage using Python Multiprocessing module. However the questions don't seem to answer the problem I have here. I am posting my analysis with the hope that some one can help me.

我看过几篇关于使用 Python 多处理模块的内存使用情况的帖子。然而，这些问题似乎并没有回答我在这里遇到的问题。我发布我的分析，希望有人可以帮助我。

Issue

问题

I am using multiprocessing to perform tasks in parallel and I noticed that the memory consumption by the worker processes grow indefinitely. I have a small standalone example that should replicate what I notice.

我使用多处理并行执行任务，我注意到工作进程的内存消耗无限增长。我有一个小的独立示例，它应该复制我注意到的内容。

import multiprocessing as mp
import time

def calculate(num):
    l = [num*num for num in range(num)]
    s = sum(l)
    del l       # delete lists as an  option
    return s

if __name__ == "__main__":
    pool = mp.Pool(processes=2)
    time.sleep(5)
    print "launching calculation"
    num_tasks = 1000
    tasks =  [pool.apply_async(calculate,(i,)) for i in range(num_tasks)]
    for f in tasks:    
        print f.get(5)
    print "calculation finished"
    time.sleep(10)
    print "closing  pool"
    pool.close()
    print "closed pool"
    print "joining pool"
    pool.join()
    print "joined pool"
    time.sleep(5)

System

系统

I am running Windows and I use the task manager to monitor the memory usage. I am running Python 2.7.6.

我正在运行 Windows 并使用任务管理器来监视内存使用情况。我正在运行 Python 2.7.6。

Observation

观察

I have summarized the memory consumption by the 2 worker processes below.

我总结了下面 2 个工作进程的内存消耗。

+---------------+----------------------+----------------------+
|  num_tasks    |  memory with del     | memory without del   |
|               | proc_1   | proc_2    | proc_1   | proc_2    |
+---------------+----------------------+----------------------+
| 1000          | 4884     | 4694      | 4892     | 4952      |
| 5000          | 5588     | 5596      | 6140     | 6268      |
| 10000         | 6528     | 6580      | 6640     | 6644      |
+---------------+----------------------+----------------------+

In the table above, I tried to change the number of tasks and observe the memory consumed at the end of all calculation and before join-ing the pool. The 'del' and 'without del' options are whether I un-comment or comment the del lline inside the calculate(num)function respectively. Before calculation, the memory consumption is around 4400.

在上表中，我尝试更改任务数量并观察所有计算结束时和join-ing之前消耗的内存pool。'del' 和 'without del' 选项分别是我是取消注释还是注释函数del l内的行calculate(num)。在计算之前，内存消耗在4400左右。

1. It looks like manually clearing out the lists results in lower memory usage for the worker processes. I thought the garbage collector would have taken care of this. Is there a way to force garbage collection?
2. It is puzzling that with increase in number of tasks, the memory usage keeps growing in both cases. Is there a way to limit the memory usage?

1. 看起来手动清除列表会导致工作进程的内存使用量降低。我认为垃圾收集器会处理这个问题。有没有办法强制垃圾收集？
2. 令人费解的是，随着任务数量的增加，两种情况下的内存使用量都在不断增长。有没有办法限制内存使用？

I have a process that is based on this example, and is meant to run long term. I observe that this worker processes are hogging up lots of memory(~4GB) after an overnight run. Doing a jointo release memory is not an option and I am trying to figure out a way without join-ing.

我有一个基于此示例的流程，旨在长期运行。我观察到这个工作进程在通宵运行后占用了大量内存（~4GB）。做一个join释放内存不是一个选项，我试图找出一种没有join-ing的方法。

This seems a little mysterious. Has anyone encountered something similar? How can I fix this issue?

这似乎有点神秘。有没有人遇到过类似的事情？我该如何解决这个问题？

采纳答案by Goutham

I did a lot of research, and couldn't find a solution to fix the problem per se. But there is a decent work around that prevents the memory blowout for a small cost, worth especially on server side long running code.

我做了很多研究，但无法找到解决问题本身的解决方案。但是有一个体面的工作可以以很小的代价防止内存爆裂，尤其是在服务器端长时间运行的代码上。

The solution essentially was to restart individual worker processes after a fixed number of tasks. The Poolclass in python takes maxtasksperchildas an argument. You can specify maxtasksperchild=1000thus limiting 1000 tasks to be run on each child process. After reaching the maxtasksperchildnumber, the pool refreshes its child processes. Using a prudent number for maximum tasks, one can balance the max memory that is consumed, with the start up cost associated with restarting back-end process. The Poolconstruction is done as :

解决方案本质上是在固定数量的任务后重新启动单个工作进程。Poolpython 中的类maxtasksperchild作为一个参数。您可以指定maxtasksperchild=1000限制在每个子进程上运行的 1000 个任务。达到该maxtasksperchild数量后，池会刷新其子进程。对最大任务使用谨慎的数字，可以平衡消耗的最大内存与与重新启动后端进程相关的启动成本。该Pool工程完成后如下：

pool = mp.Pool(processes=2,maxtasksperchild=1000)

I am putting my full solution here so it can be of use to others!

我将我的完整解决方案放在这里，以便对其他人有用！

import multiprocessing as mp
import time

def calculate(num):
    l = [num*num for num in range(num)]
    s = sum(l)
    del l       # delete lists as an  option
    return s

if __name__ == "__main__":

    # fix is in the following line #
    pool = mp.Pool(processes=2,maxtasksperchild=1000)

    time.sleep(5)
    print "launching calculation"
    num_tasks = 1000
    tasks =  [pool.apply_async(calculate,(i,)) for i in range(num_tasks)]
    for f in tasks:    
        print f.get(5)
    print "calculation finished"
    time.sleep(10)
    print "closing  pool"
    pool.close()
    print "closed pool"
    print "joining pool"
    pool.join()
    print "joined pool"
    time.sleep(5)

回答by Mike

One potential problem here is that results could be coming back in any order, but because you're reading them in order, it has to store all the results coming back from the processes in memory. The higher num_tasksis, the more results it potentially has to store in memory waiting for your for f in tasksloop to process the result.

这里的一个潜在问题是结果可能以任何顺序返回，但由于您正在按顺序读取它们，因此它必须将从进程返回的所有结果存储在内存中。num_tasks越高，它可能需要在内存中存储的结果越多，等待您的for f in tasks循环来处理结果。

In the worst case, the results are calculated in exactly reverse order. In that case, all the results must be held by the multiprocessing module in memory for you before your for f in tasksloop will start processing anything.

在最坏的情况下，结果以完全相反的顺序计算。在这种情况下，所有结果都必须由 multiprocessing 模块保存在内存中，然后您的for f in tasks循环将开始处理任何内容。

It does seem like the amount of memory they're using is higher than I'd expect in this case though (more than it should be just for storing the 1000-10000 numbers returned by the calculate() function), but maybe there's just a high constant overhead per worker result that's stored up.

在这种情况下，他们使用的内存量似乎确实比我预期的要高（比仅用于存储 calculate() 函数返回的 1000-10000 个数字应该多），但也许只是存储的每个工人结果的高恒定开销。

Have you tried specifying the callbackparameter to apply_async, so you can process results immediately as they're completed, or using imap_unordered, so it can give you back results as soon as they're ready?

您是否尝试过将回调参数指定给apply_async，以便您可以在结果完成后立即处理结果，或者使用imap_unordered，以便在结果准备好后立即将结果返回给您？