列出 ipython 和 jupyter 中的内存使用情况

声明:本页面是StackOverFlow热门问题的中英对照翻译,遵循CC BY-SA 4.0协议,如果您需要使用它,必须同样遵循CC BY-SA许可,注明原文地址和作者信息,同时你必须将它归于原作者(不是我):StackOverFlow 原文地址: http://stackoverflow.com/questions/40993626/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me): StackOverFlow

提示:将鼠标放在中文语句上可以显示对应的英文。显示中英文
时间:2020-08-20 00:18:45  来源:igfitidea点击:

list memory usage in ipython and jupyter

pythonnumpymemoryipythonjupyter-notebook

提问by Rho Phi

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

我有几个(几乎十个) Gb 的内存被 ipython 内核占用。我认为这是来自我可能在某些操作期间产生的大对象(矩阵、列表、numpy 数组,...),现在我不再需要了。

I would like to list all of the objects I have definedand sort themby their memory footprint. Is there a simple way to do that? For certain types there is nbytesmethod, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

我想列出我定义的所有对象并按它们的内存占用量对它们进行排序。有没有一种简单的方法可以做到这一点?对于某些类型有nbytes方法,但不是所有的......所以我正在寻找一种通用的方法来列出我制作的所有对象及其内存占用。

回答by Abdou

Assuming that you are using ipythonor jupyter, you will need to do a little bit of work to get a list all of the objects youhave defined. That means taking everything available in globals()and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

假设您正在使用ipythonjupyter,您需要做一些工作来获取定义的所有对象的列表。这意味着要提供一切globals()和剔除这些对象modulesbuiltinsipython objects,等等。一旦你确定你有这些对象,那么你就可以继续与抢夺它们的大小sys.getsizeof。这可以总结如下:

import sys

# These are the usual ipython objects, including this one you are creating
ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']

# Get a sorted list of the objects and their sizes
sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True)

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeofwill be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeofadds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

请记住,对于 python 对象(使用 python 的内置函数创建的对象),sys.getsizeof将非常准确。但是对于使用第三方库创建的对象,它可能有点不准确。此外,请注意,sys.getsizeof如果对象由垃圾收集器管理,则会增加额外的垃圾收集器开销。所以,有些东西看起来可能比实际重一些。

As a side note, numpy's .nbytesmethod can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

作为旁注,numpy.nbytes方法可能有些误导,因为它不包括数组对象的非元素属性消耗的内存。

I hope this helps.

我希望这有帮助。