Python 无法分配具有形状和数据类型的数组
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/57507832/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Unable to allocate array with shape and data type
提问by Martin Bri?iak
I'm facing an issue with allocating huge arrays in numpy on Ubuntu 18 while not facing the same issue on MacOS.
我在 Ubuntu 18 上的 numpy 中遇到了分配大数组的问题，而在 MacOS 上没有遇到同样的问题。
I am trying to allocate memory for a numpy array with shape (156816, 36, 53806)with
我想一个numpy的阵列形状分配内存(156816, 36, 53806)使用
np.zeros((156816, 36, 53806), dtype='uint8')
and while I'm getting an error on Ubuntu OS
当我在 Ubuntu 操作系统上遇到错误时
>>> import numpy as np
>>> np.zeros((156816, 36, 53806), dtype='uint8')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
numpy.core._exceptions.MemoryError: Unable to allocate array with shape (156816, 36, 53806) and data type uint8
I'm not getting it on MacOS:
我没有在 MacOS 上得到它：
>>> import numpy as np
>>> np.zeros((156816, 36, 53806), dtype='uint8')
array([[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
...,
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]],
[[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
...,
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0],
[0, 0, 0, ..., 0, 0, 0]]], dtype=uint8)
I've read somewhere that np.zerosshouldn't be really allocating the whole memory needed for the array, but only for the non-zero elements. Even though the Ubuntu machine has 64gb of memory, while my MacBook Pro has only 16gb.
我在某处读到np.zeros不应该真正分配数组所需的整个内存，而只为非零元素分配。即使 Ubuntu 机器有 64GB 的内存，而我的 MacBook Pro 只有 16GB。
versions:
版本：
Ubuntu
os -> ubuntu mate 18
python -> 3.6.8
numpy -> 1.17.0
mac
os -> 10.14.6
python -> 3.6.4
numpy -> 1.17.0
PS: also failed on Google Colab
PS：在 Google Colab 上也失败了
回答by Iguananaut
This is likely due to your system's overcommit handlingmode.
这可能是由于您系统的过量使用处理模式造成的。
In the default mode, 0,
在默认模式下0，
Heuristic overcommit handling. Obvious overcommits of address space are refused. Used for a typical system. It ensures a seriously wild allocation fails while allowing overcommit to reduce swap usage. root is allowed to allocate slightly more memory in this mode. This is the default.
启发式过量使用处理。明显的地址空间过度使用被拒绝。用于典型系统。它确保严重的疯狂分配失败，同时允许过度使用以减少交换使用。在这种模式下允许 root 分配更多的内存。这是默认设置。
The exact heuristic used is not well explained here, but this is discussed more on Linux over commit heuristicand on this page.
此处没有很好地解释所使用的确切启发式方法，但在Linux 上通过提交启发式方法和此页面上对此进行了更多讨论 。
You can check your current overcommit mode by running
您可以通过运行来检查当前的过量使用模式
$ cat /proc/sys/vm/overcommit_memory
0
In this case you're allocating
在这种情况下，您正在分配
>>> 156816 * 36 * 53806 / 1024.0**3
282.8939827680588
~282 GB, and the kernel is saying well obviously there's no way I'm going to be able to commit that many physical pages to this, and it refuses the allocation.
~282 GB，内核说得很好，显然我无法将那么多物理页面提交给它，并且它拒绝分配。
If (as root) you run:
如果（以 root 身份）运行：
$ echo 1 > /proc/sys/vm/overcommit_memory
This will enable "always overcommit" mode, and you'll find that indeed the system will allow you to make the allocation no matter how large it is (within 64-bit memory addressing at least).
这将启用“始终过量使用”模式，您会发现系统确实允许您进行分配，无论它有多大（至少在 64 位内存寻址范围内）。
I tested this myself on a machine with 32 GB of RAM. With overcommit mode 0I also got a MemoryError, but after changing it back to 1it works:
我自己在具有 32 GB RAM 的机器上对此进行了测试。在过量使用模式下，0我也得到了一个MemoryError，但在将其改回后1可以工作：
>>> import numpy as np
>>> a = np.zeros((156816, 36, 53806), dtype='uint8')
>>> a.nbytes
303755101056
You can then go ahead and write to any location within the array, and the system will only allocate physical pages when you explicitly write to that page. So you can use this, with care, for sparse arrays.
然后，您可以继续写入阵列中的任何位置，系统只会在您明确写入该页面时分配物理页面。所以你可以小心地将它用于稀疏数组。
回答by recurseuntilfor
I had this same problem on Window's and came across this solution. So if someone comes across this problem in Windows the solution for me was to increase the pagefilesize, as it was a Memory overcommitment problem for me too.
我在 Window 上遇到了同样的问题，并遇到了这个解决方案。因此，如果有人在 Windows 中遇到此问题，我的解决方案是增加页面文件大小，因为这对我来说也是内存过量使用的问题。
Windows 8
视窗 8
- On the Keyboard Press the WindowsKey + X then click System in the popup menu
- Tap or click Advanced system settings. You might be asked for an admin password or to confirm your choice
- On the Advanced tab, under Performance, tap or click Settings.
- Tap or click the Advanced tab, and then, under Virtual memory, tap or click Change
- Clear the Automatically manage paging file size for all drives check box.
- Under Drive [Volume Label], tap or click the drive that contains the paging file you want to change
- Tap or click Custom size, enter a new size in megabytes in the initial size (MB) or Maximum size (MB) box, tap or click Set, and then tap or click OK
- Reboot your system
- 在键盘上按 Windows 键 + X，然后在弹出菜单中单击系统
- 点击或单击高级系统设置。系统可能会要求您输入管理员密码或确认您的选择
- 在“高级”选项卡上的“性能”下，点击或单击“设置”。
- 点击或单击“高级”选项卡，然后在“虚拟内存”下，点击或单击“更改”
- 清除自动管理所有驱动器的分页文件大小复选框。
- 在驱动器 [卷标] 下，点击或单击包含要更改的分页文件的驱动器
- 点击或单击自定义大小，在初始大小 (MB) 或最大大小 (MB) 框中输入以兆字节为单位的新大小，点击或单击设置，然后点击或单击确定
- 重新启动系统
Windows 10
视窗 10
- Press the Windows key
- Type SystemPropertiesAdvanced
- Click Run as administrator
- Click Settings
- Select the Advanced tab
- Select Change...
- Uncheck Automatically managing paging file size for all drives
- Then select Custom size and fill in the appropriate size
- Press Set then press OK then exit from the Virtual Memory, Performance Options, and System Properties Dialog
- Reboot your system
- 按 Windows 键
- 类型 SystemPropertiesAdvanced
- 点击以管理员身份运行
- 单击设置
- 选择高级选项卡
- 选择更改...
- 取消选中自动管理所有驱动器的分页文件大小
- 然后选择Custom size并填写合适的尺寸
- 按 Set 然后按 OK 然后退出 Virtual Memory、Performance Options 和 System Properties 对话框
- 重新启动系统
Note: I did not have the enough memory on my system for the ~282GB in this example but for my particular case this worked.
注意：在本示例中，我的系统上没有足够的内存用于 ~282GB，但对于我的特殊情况，这是有效的。
EDIT
编辑
From herethe suggested recommendations for page file size:
从这里对页面文件大小的建议建议：
There is a formula for calculating the correct pagefile size. Initial size is one and a half (1.5) x the amount of total system memory. Maximum size is three (3) x the initial size. So let's say you have 4 GB (1 GB = 1,024 MB x 4 = 4,096 MB) of memory. The initial size would be 1.5 x 4,096 = 6,144 MB and the maximum size would be 3 x 6,144 = 18,432 MB.
有一个计算正确页面文件大小的公式。初始大小是系统总内存量的二分之一 (1.5) x。最大尺寸为三 (3) x 初始尺寸。因此，假设您有 4 GB（1 GB = 1,024 MB x 4 = 4,096 MB）的内存。初始大小为 1.5 x 4,096 = 6,144 MB，最大大小为 3 x 6,144 = 18,432 MB。
Some things to keep in mind from here:
从这里要记住的一些事情：
However, this does not take into consideration other important factors and system settings that may be unique to your computer. Again, let Windows choose what to use instead of relying on some arbitrary formula that worked on a different computer.
但是，这并未考虑您的计算机可能独有的其他重要因素和系统设置。同样，让 Windows 选择要使用的内容，而不是依赖某些在不同计算机上运行的任意公式。
Also:
还：
Increasing page file size may help prevent instabilities and crashing in Windows. However, a hard drive read/write times are much slower than what they would be if the data were in your computer memory. Having a larger page file is going to add extra work for your hard drive, causing everything else to run slower. Page file size should only be increased when encountering out-of-memory errors, and only as a temporary fix. A better solution is to adding more memory to the computer.
增加页面文件大小可能有助于防止 Windows 中的不稳定和崩溃。但是，硬盘驱动器的读/写时间比数据在计算机内存中时慢得多。拥有更大的页面文件将为您的硬盘增加额外的工作，导致其他一切运行速度变慢。仅在遇到内存不足错误时才应增加页面文件大小，并且只能作为临时修复。更好的解决方案是向计算机添加更多内存。
回答by kotchwane
I came across this problem on Windows too. The solution for me was to switch from a 32-bit to a 64-bit version of Python. Indeed, a 32-bit software, like a 32-bit CPU, can adress a maximum of 4?GBof RAM (2^32). So if you have more than 4 GB of RAM, a 32-bit version cannot take advantage of it.
我在 Windows 上也遇到了这个问题。我的解决方案是从 32 位版本切换到 64 位版本的 Python。事实上，一个 32 位的软件，就像一个 32 位的 CPU，最多可以处理 4?GB的 RAM (2^32)。因此，如果您拥有超过 4 GB 的 RAM，则 32 位版本无法利用它。
With a 64-bit version of Python (the one labeled x86-64in the download page), the issue disappeared.
使用 64 位版本的 Python（下载页面中标记为x86-64 的版本），问题消失了。
You can check which version you have by entering the interpreter. I, with a 64-bit version, now have:
Python 3.7.5rc1 (tags/v3.7.5rc1:4082f600a5, Oct 1 2019, 20:28:14) [MSC v.1916 64 bit (AMD64)], where [MSC v.1916 64 bit (AMD64)] means "64-bit Python".
您可以通过输入解释器来检查您的版本。我，使用 64 位版本，现在有：
Python 3.7.5rc1 (tags/v3.7.5rc1:4082f600a5, Oct 1 2019, 20:28:14) [MSC v.1916 64 bit (AMD64)]，其中 [MSC v.1916 64 位 (AMD64)] 表示“64 位 Python”。
Note: as of the time of this writing (May 2020), matplotlib is not availableon python39, so I recommand installing python37, 64 bits.
注：截至发稿（五月2020年）的时候，matplotlib是不可用的python39，所以我安装推荐python37，64位。
Sources :
来源：
回答by Pragya Agrawal
In my case, adding a dtype attribute changed dtype of the array to a smaller type(from float64 to uint8), decreasing array size enough to not throw MemoryError in Windows(64 bit).
在我的情况下，添加 dtype 属性将数组的 dtype 更改为较小的类型（从 float64 到 uint8），减少数组大小足以在 Windows（64 位）中不抛出 MemoryError。
from
从
mask = np.zeros(edges.shape)
to
到
mask = np.zeros(edges.shape,dtype='uint8')
回答by Antonio Moreno Martín
Sometimes, this error pops up because of the kernel has reached its limit. Try to restart the kernel redo the necessary steps.
有时，由于内核已达到极限，会弹出此错误。尝试重新启动内核重做必要的步骤。
