python如何获得iobytes分配的内存长度？

python how to get iobytes allocated memory length?

提问by user4046642

This is the code i am using to test the memory allocation

这是我用来测试内存分配的代码

import pycurl
import io


url = "http://www.stackoverflow.com"
buf = io.BytesIO()


print(len(buf.getvalue()))   #here i am getting 0 as length


c = pycurl.Curl()
c.setopt(c.URL, url)
c.setopt(c.CONNECTTIMEOUT, 10)
c.setopt(c.TIMEOUT, 10)
c.setopt(c.ENCODING, 'gzip')
c.setopt(c.FOLLOWLOCATION, True)
c.setopt(c.IPRESOLVE, c.IPRESOLVE_V4)
c.setopt(c.USERAGENT, 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:8.0) Gecko/20100101 Firefox/8.0')
c.setopt(c.WRITEFUNCTION, buf.write)
c.perform()
c.close()

print(len(buf.getvalue()))    #here length of the dowloaded file


print(buf.getvalue())
buf.close()

How to get the allocated buffer/memory length by BytesIO ? what am i doing wrong here ? python doesn't allocate fixed buffer length ?

如何通过 BytesIO 获取分配的缓冲区/内存长度？我在这里做错了什么？python 不分配固定的缓冲区长度？

采纳答案by mhawke

I am not sure what you mean by allocated buffer/memory length, but if you want the length of the user data stored in the BytesIOobject you can do

我不确定分配的缓冲区/内存长度是什么意思，但是如果您想要存储在BytesIO对象中的用户数据的长度，您可以这样做

>>> bio = io.BytesIO()
>>> bio.getbuffer().nbytes
0
>>> bio.write(b'here is some data')
17
>>> bio.getbuffer().nbytes
17

But this seems equivalent to the len(buf.getvalue())that you are currently using.

但这似乎等同于len(buf.getvalue())您当前使用的。

The actual size of the BytesIO object can be found using sys.getsizeof():

可以使用sys.getsizeof()以下命令找到 BytesIO 对象的实际大小：

>>> bio = io.BytesIO()
>>> sys.getsizeof(bio)
104

Or you could be nasty and call __sizeof__()directly (which is like sys.getsizeof()but without garbage collector overhead applicable to the object):

或者你可能很讨厌并__sizeof__()直接调用（这就像sys.getsizeof()但没有适用于对象的垃圾收集器开销）：

>>> bio = io.BytesIO()
>>> bio.__sizeof__()
72

Memory for BytesIOis allocated as required, and some buffering does take place:

BytesIO根据需要分配内存，并且确实会发生一些缓冲：

>>> bio = io.BytesIO()
>>> for i in range(20):
...     _=bio.write(b'a')
...     print(bio.getbuffer().nbytes, sys.getsizeof(bio), bio.__sizeof__())
...
1 106 74
2 106 74
3 108 76
4 108 76
5 110 78
6 110 78
7 112 80
8 112 80
9 120 88
10 120 88
11 120 88
12 120 88
13 120 88
14 120 88
15 120 88
16 120 88
17 129 97
18 129 97
19 129 97
20 129 97

回答by rth

io.BytesIO()returns a standard file object which has function tell(). It reports the current descriptor position and does not copy the whole buffer out to compute total size as len(bio.getvalue())of bio.getbuffer().nbytes. It is a very fast and simple method to get the exact size of used memory in the buffer object.

io.BytesIO()返回一个具有函数的标准文件对象tell()。它报告将当前描述位置，不会复制整个缓冲区以计算总大小len(bio.getvalue())的bio.getbuffer().nbytes。这是获取缓冲区对象中已用内存的确切大小的一种非常快速且简单的方法。

I posted an example code and a more detailed answer here

我在这里发布了示例代码和更详细的答案

回答by ti7

You can also use tracemallocto get indirect information about the size of objects, by wrapping memory events in tracemalloc.get_traced_memory()

您还可以tracemalloc通过将内存事件包装在tracemalloc.get_traced_memory()

Do note that active threads (if any) and side effects of your program will affect the output, but it may also be more representative of the real memory cost if many samples are taken, as shown below.

请注意，活动线程（如果有）和程序的副作用会影响输出，但如果采样很多，它也可能更能代表实际内存成本，如下所示。

>>> import tracemalloc
>>> from io import BytesIO
>>> tracemalloc.start()
>>>
>>> memory_traces = []
>>>
>>> with BytesIO() as bytes_fh:
...     # returns (current memory usage, peak memory usage)
        # ..but only since calling .start()
...     memory_traces.append(tracemalloc.get_traced_memory())
...     bytes_fh.write(b'a' * (1024**2))  # create 1MB of 'a'
...     memory_traces.append(tracemalloc.get_traced_memory())
...
1048576
>>> print("used_memory = {}b".format(memory_traces[1][0] - memory_traces[0][0]))
used_memory = 1048870b
>>> 1048870 - 1024**2  # show small overhead
294