在 Python 中释放内存

Releasing memory in Python

提问by Jared

I have a few related questions regarding memory usage in the following example.

在以下示例中，我有一些有关内存使用的相关问题。

1. If I run in the interpreter,

   foo = ['bar' for _ in xrange(10000000)]
   

   the real memory used on my machine goes up to 80.9mb. I then,

   del foo
   

   real memory goes down, but only to 30.4mb. The interpreter uses 4.4mbbaseline so what is the advantage in not releasing 26mbof memory to the OS? Is it because Python is "planning ahead", thinking that you may use that much memory again?

2. Why does it release 50.5mbin particular - what is the amount that is released based on?

3. Is there a way to force Python to release all the memory that was used (if you know you won't be using that much memory again)?

1. 如果我在解释器中运行，

   foo = ['bar' for _ in xrange(10000000)]
   

   我机器上使用的实际内存高达80.9mb. 那我，

   del foo
   

   真实内存下降，但仅限于30.4mb. 解释器使用4.4mb基线，那么不26mb向操作系统释放内存的优势是什么？是不是因为 Python 正在“提前计划”，认为您可能会再次使用那么多内存？

2. 为什么它会50.5mb特别释放-释放的数量是多少？

3. 有没有办法强制 Python 释放所有使用过的内存（如果你知道你不会再使用那么多内存）？

NOTEThis question is different from How can I explicitly free memory in Python?because this question primarily deals with the increase of memory usage from baseline even after the interpreter has freed objects via garbage collection (with use of gc.collector not).

注意这个问题与如何在 Python 中显式释放内存不同？因为即使在解释器通过垃圾收集（使用gc.collect或不使用）释放对象之后，这个问题也主要处理从基线开始的内存使用量的增加。

采纳答案by Eryk Sun

Memory allocated on the heap can be subject to high-water marks. This is complicated by Python's internal optimizations for allocating small objects (PyObject_Malloc) in 4 KiB pools, classed for allocation sizes at multiples of 8 bytes -- up to 256 bytes (512 bytes in 3.3). The pools themselves are in 256 KiB arenas, so if just one block in one pool is used, the entire 256 KiB arena will not be released. In Python 3.3 the small object allocator was switched to using anonymous memory maps instead of the heap, so it should perform better at releasing memory.

在堆上分配的内存可能会受到高水位标记的影响。PythonPyObject_Malloc在 4 KiB 池中分配小对象 ( )的内部优化使这变得复杂，分配大小按 8 字节的倍数分类——最多 256 字节（3.3 中为 512 字节）。池本身位于 256 KiB 的竞技场中，因此如果仅使用一个池中的一个区块，则不会释放整个 256 KiB 的竞技场。在 Python 3.3 中，小对象分配器被切换为使用匿名内存映射而不是堆，因此它在释放内存方面应该表现得更好。

Additionally, the built-in types maintain freelists of previously allocated objects that may or may not use the small object allocator. The inttype maintains a freelist with its own allocated memory, and clearing it requires calling PyInt_ClearFreeList(). This can be called indirectly by doing a full gc.collect.

此外，内置类型维护先前分配的对象的空闲列表，这些对象可能会或可能不会使用小对象分配器。该int类型使用自己分配的内存维护一个空闲列表，清除它需要调用PyInt_ClearFreeList(). 这可以通过执行 full 间接调用gc.collect。

Try it like this, and tell me what you get. Here's the link for psutil.Process.memory_info.

像这样尝试，然后告诉我你得到了什么。这是psutil.Process.memory_info的链接。

import os
import gc
import psutil

proc = psutil.Process(os.getpid())
gc.collect()
mem0 = proc.get_memory_info().rss

# create approx. 10**7 int objects and pointers
foo = ['abc' for x in range(10**7)]
mem1 = proc.get_memory_info().rss

# unreference, including x == 9999999
del foo, x
mem2 = proc.get_memory_info().rss

# collect() calls PyInt_ClearFreeList()
# or use ctypes: pythonapi.PyInt_ClearFreeList()
gc.collect()
mem3 = proc.get_memory_info().rss

pd = lambda x2, x1: 100.0 * (x2 - x1) / mem0
print "Allocation: %0.2f%%" % pd(mem1, mem0)
print "Unreference: %0.2f%%" % pd(mem2, mem1)
print "Collect: %0.2f%%" % pd(mem3, mem2)
print "Overall: %0.2f%%" % pd(mem3, mem0)

Output:

输出：

Allocation: 3034.36%
Unreference: -752.39%
Collect: -2279.74%
Overall: 2.23%

Edit:

编辑：

I switched to measuring relative to the process VM size to eliminate the effects of other processes in the system.

我切换到相对于进程 VM 大小的测量，以消除系统中其他进程的影响。

The C runtime (e.g. glibc, msvcrt) shrinks the heap when contiguous free space at the top reaches a constant, dynamic, or configurable threshold. With glibc you can tune this with mallopt(M_TRIM_THRESHOLD). Given this, it isn't surprising if the heap shrinks by more -- even a lot more -- than the block that you free.

当顶部的连续可用空间达到恒定、动态或可配置的阈值时，C 运行时（例如 glibc、msvcrt）会缩小堆。使用 glibc，您可以使用mallopt(M_TRIM_THRESHOLD) 进行调整。鉴于此，如果堆比您缩小的块收缩得更多——甚至更多——也就不足为奇了free。

In 3.x rangedoesn't create a list, so the test above won't create 10 million intobjects. Even if it did, the inttype in 3.x is basically a 2.x long, which doesn't implement a freelist.

在 3.xrange中不创建列表，因此上面的测试不会创建 1000 万个int对象。即使有，int3.x 中的类型基本上是 2.x long，它没有实现自由列表。

回答by abarnert

I'm guessing the question you really care about here is:

我猜你在这里真正关心的问题是：

Is there a way to force Python to release all the memory that was used (if you know you won't be using that much memory again)?

有没有办法强制 Python 释放所有使用过的内存（如果你知道你不会再使用那么多内存）？

No, there is not. But there is an easy workaround: child processes.

不，那里没有。但是有一个简单的解决方法：子进程。

If you need 500MB of temporary storage for 5 minutes, but after that you need to run for another 2 hours and won't touch that much memory ever again, spawn a child process to do the memory-intensive work. When the child process goes away, the memory gets released.

如果您需要 500MB 的临时存储空间 5 分钟，但之后您需要再运行 2 小时并且不会再接触那么多内存，则生成一个子进程来完成内存密集型工作。当子进程消失时，内存被释放。

This isn't completely trivial and free, but it's pretty easy and cheap, which is usually good enough for the trade to be worthwhile.

这并不是完全琐碎和免费的，但它非常简单且便宜，通常足以让交易变得有价值。

First, the easiest way to create a child process is with concurrent.futures(or, for 3.1 and earlier, the futuresbackport on PyPI):

首先，创建子进程的最简单方法是使用concurrent.futures（或者，对于 3.1 及更早版本，futures使用 PyPI 上的backport）：

with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor:
    result = executor.submit(func, *args, **kwargs).result()

If you need a little more control, use the multiprocessingmodule.

如果您需要更多控制，请使用该multiprocessing模块。

The costs are:

费用是：

• Process startup is kind of slow on some platforms, notably Windows. We're talking milliseconds here, not minutes, and if you're spinning up one child to do 300 seconds' worth of work, you won't even notice it. But it's not free.
• If the large amount of temporary memory you use really is large, doing this can cause your main program to get swapped out. Of course you're saving time in the long run, because that if that memory hung around forever it would have to lead to swapping at some point. But this can turn gradual slowness into very noticeable all-at-once (and early) delays in some use cases.
• Sending large amounts of data between processes can be slow. Again, if you're talking about sending over 2K of arguments and getting back 64K of results, you won't even notice it, but if you're sending and receiving large amounts of data, you'll want to use some other mechanism (a file, mmapped or otherwise; the shared-memory APIs in multiprocessing; etc.).
• Sending large amounts of data between processes means the data have to be pickleable (or, if you stick them in a file or shared memory, struct-able or ideally ctypes-able).

• 在某些平台上，特别是 Windows，进程启动有点慢。我们在这里谈论的是毫秒，而不是分钟，如果你让一个孩子做 300 秒的工作，你甚至不会注意到它。但它不是免费的。
• 如果您使用的大量临时内存确实很大，那么这样做可能会导致您的主程序被换出。当然，从长远来看，您可以节省时间，因为如果该内存永远存在，则必须在某些时候进行交换。但这可能会在某些用例中将逐渐缓慢变成非常明显的一次性（和早期）延迟。
• 在进程之间发送大量数据可能会很慢。同样，如果您正在谈论发送超过 2K 的参数并返回 64K 的结果，您甚至不会注意到它，但是如果您正在发送和接收大量数据，您将需要使用其他一些机制（文件，mmapped 或其他；共享内存 API multiprocessing；等）。
• 在进程之间发送大量数据意味着数据必须是可腌制的（或者，如果您将它们保存在文件或共享内存中，struct则可以或理想情况下ctypes可以）。

回答by abarnert

eryksun has answered question #1, and I've answered question #3 (the original #4), but now let's answer question #2:

eryksun 已经回答了问题 #1，我已经回答了问题 #3（原来的 #4），但现在让我们回答问题 #2：

Why does it release 50.5mb in particular - what is the amount that is released based on?

为什么它特别释放 50.5mb - 释放的数量是多少？

What it's based on is, ultimately, a whole series of coincidences inside Python and mallocthat are very hard to predict.

它最终基于 Python 内部的一系列巧合，而且malloc这些巧合很难预测。

First, depending on how you're measuring memory, you may only be measuring pages actually mapped into memory. In that case, any time a page gets swapped out by the pager, memory will show up as "freed", even though it hasn't been freed.

首先，根据您测量内存的方式，您可能只测量实际映射到内存中的页面。在这种情况下，每当页面被寻呼机换出时，内存将显示为“已释放”，即使它尚未被释放。

Or you may be measuring in-use pages, which may or may not count allocated-but-never-touched pages (on systems that optimistically over-allocate, like linux), pages that are allocated but tagged MADV_FREE, etc.

或者您可能正在测量使用中的页面，这些页面可能会或可能不会计算已分配但从未接触过的页面（在乐观过度分配的系统上，例如 linux）、已分配但已标记的页面MADV_FREE等。

If you really are measuring allocated pages (which is actually not a very useful thing to do, but it seems to be what you're asking about), and pages have really been deallocated, two circumstances in which this can happen: Either you've used brkor equivalent to shrink the data segment (very rare nowadays), or you've used munmapor similar to release a mapped segment. (There's also theoretically a minor variant to the latter, in that there are ways to release part of a mapped segment—e.g., steal it with MAP_FIXEDfor a MADV_FREEsegment that you immediately unmap.)

如果你真的在测量分配的页面（这实际上不是一件非常有用的事情，但它似乎是你要问的），并且页面真的被释放了，这可能发生在两种情况下：要么你已经使用brk或等效于缩小数据段（现在非常罕见），或者您已经使用munmap或类似方法来释放映射段。（理论上，后者还有一个次要的变体，因为有一些方法可以释放映射段的一部分——例如，MAP_FIXED为一个MADV_FREE你立即取消映射的段窃取它。）

But most programs don't directly allocate things out of memory pages; they use a malloc-style allocator. When you call free, the allocator can only release pages to the OS if you just happen to be freeing the last live object in a mapping (or in the last N pages of the data segment). There's no way your application can reasonably predict this, or even detect that it happened in advance.

但是大多数程序并不直接从内存页中分配东西；他们使用malloc-style 分配器。当您调用 时free，如果您恰好free在映射中（或数据段的最后 N 个页面）中访问最后一个活动对象，则分配器只能将页面释放到操作系统。您的应用程序无法合理地预测这一点，甚至无法提前检测到它发生了。

CPython makes this even more complicated—it has a custom 2-level object allocator on top of a custom memory allocator on top of malloc. (See the source commentsfor a more detailed explanation.) And on top of that, even at the C API level, much less Python, you don't even directly control when the top-level objects are deallocated.

CPython 使这变得更加复杂——它在malloc. （请参阅源注释以获得更详细的解释。）最重要的是，即使在 C API 级别，更不用说 Python，您甚至无法直接控制何时释放顶级对象。

So, when you release an object, how do you know whether it's going to release memory to the OS? Well, first you have to know that you've released the last reference (including any internal references you didn't know about), allowing the GC to deallocate it. (Unlike other implementations, at least CPython will deallocate an object as soon as it's allowed to.) This usually deallocates at least two things at the next level down (e.g., for a string, you're releasing the PyStringobject, and the string buffer).

那么，当你释放一个对象时，你怎么知道它是否会向操作系统释放内存？好吧，首先您必须知道您已经释放了最后一个引用（包括您不知道的任何内部引用），从而允许 GC 释放它。（与其他实现不同，至少 CPython 会在允许的情况下尽快释放对象。）这通常会在下一级释放至少两件事（例如，对于字符串，您正在释放PyString对象，以及字符串缓冲区）。

If you dodeallocate an object, to know whether this causes the next level down to deallocate a block of object storage, you have to know the internal state of the object allocator, as well as how it's implemented. (It obviously can't happen unless you're deallocating the last thing in the block, and even then, it may not happen.)

如果你确实释放了一个对象，要知道这是否会导致下一级释放对象存储块，你必须知道对象分配器的内部状态，以及它是如何实现的。（除非您释放块中的最后一件事，否则它显然不会发生，即使如此，它也可能不会发生。）

If you dodeallocate a block of object storage, to know whether this causes a freecall, you have to know the internal state of the PyMem allocator, as well as how it's implemented. (Again, you have to be deallocating the last in-use block within a malloced region, and even then, it may not happen.)

如果您确实释放了一个对象存储块，要知道这是否会导致free调用，您必须知道 PyMem 分配器的内部状态，以及它是如何实现的。（同样，您必须释放malloced 区域内最后一个正在使用的块，即使如此，它也可能不会发生。）

If you dofreea malloced region, to know whether this causes an munmapor equivalent (or brk), you have to know the internal state of the malloc, as well as how it's implemented. And this one, unlike the others, is highly platform-specific. (And again, you generally have to be deallocating the last in-use mallocwithin an mmapsegment, and even then, it may not happen.)

如果你做free一个malloced 区域，要知道这是否导致 anmunmap或等价物（或brk），你必须知道 的内部状态malloc，以及它是如何实现的。与其他的不同，这个是高度特定于平台的。（同样，您通常必须取消分配段malloc内最后一个使用中的分配mmap，即使如此，它也可能不会发生。）

So, if you want to understand why it happened to release exactly 50.5mb, you're going to have to trace it from the bottom up. Why did mallocunmap 50.5mb worth of pages when you did those one or more freecalls (for probably a bit more than 50.5mb)? You'd have to read your platform's malloc, and then walk the various tables and lists to see its current state. (On some platforms, it may even make use of system-level information, which is pretty much impossible to capture without making a snapshot of the system to inspect offline, but luckily this isn't usually a problem.) And then you have to do the same thing at the 3 levels above that.

所以，如果你想了解为什么它恰好释放了 50.5mb，你将不得不自下而上地追踪它。为什么在malloc执行一次或多次free调用时取消映射 50.5mb 的页面（可能超过 50.5mb）？您必须阅读平台的malloc，然后遍历各种表格和列表以查看其当前状态。（在某些平台上，它甚至可能利用系统级信息，如果不制作系统快照以进行离线检查，几乎不可能捕获这些信息，但幸运的是，这通常不是问题。）然后你必须在上面的 3 个级别上做同样的事情。

So, the only useful answer to the question is "Because."

所以，这个问题唯一有用的答案是“因为”。

Unless you're doing resource-limited (e.g., embedded) development, you have no reason to care about these details.

除非您正在进行资源受限（例如，嵌入式）开发，否则您没有理由关心这些细节。

And if you aredoing resource-limited development, knowing these details is useless; you pretty much have to do an end-run around all those levels and specifically mmapthe memory you need at the application level (possibly with one simple, well-understood, application-specific zone allocator in between).

而如果你在做资源有限的开发，知道这些细节是没有用的；您几乎必须围绕所有这些级别，特别mmap是您在应用程序级别所需的内存（可能在两者之间使用一个简单的、易于理解的、特定于应用程序的区域分配器）进行最终运行。

回答by de20ce

First, you may want to install glances:

首先，您可能需要安装 Glances：

sudo apt-get install python-pip build-essential python-dev lm-sensors 
sudo pip install psutil logutils bottle batinfo https://bitbucket.org/gleb_zhulik/py3sensors/get/tip.tar.gz zeroconf netifaces pymdstat influxdb elasticsearch potsdb statsd pystache docker-py pysnmp pika py-cpuinfo bernhard
sudo pip install glances

Then run it in the terminal!

然后在终端中运行它！

glances

In your Python code, add at the begin of the file, the following:

在您的 Python 代码中，在文件开头添加以下内容：

import os
import gc # Garbage Collector

After using the "Big" variable (for example: myBigVar) for which, you would like to release memory, write in your python code the following:

在使用“Big”变量（例如：myBigVar）之后，你想释放内存，在你的python代码中写下以下内容：

del myBigVar
gc.collect()

In another terminal, run your python code and observe in the "glances" terminal, how the memory is managed in your system!

在另一个终端中，运行您的 python 代码并在“glances”终端中观察系统中的内存是如何管理的！

Good luck!

祝你好运！

P.S. I assume you are working on a Debian or Ubuntu system

PS 我假设您正在使用 Debian 或 Ubuntu 系统