C++ 在C++中创建过大的数组，如何解决？

Create too large array in C++, how to solve?

提问by Kingfisher Phuoc

Recently, I work in C++ and I have to create a array[60.000][60.000]. However, i cannot create this array because it's too large. I tried float **arrayor even static float arraybut nothing is good. Does anyone have an ideas? Thanks for your helps!

最近，我在 C++ 中工作，我必须创建一个array[60.000][60.000]. 但是，我无法创建这个数组，因为它太大了。我试过float **array，甚至static float array但没有什么是好的。有没有人有想法？感谢您的帮助！

回答by Mysticial

A matrix of size 60,000 x 60,000has 3,600,000,000elements.

大小矩阵60,000 x 60,000具有3,600,000,000元素。

You're using type floatso it becomes:

您正在使用类型，float因此它变为：

60,000 x 60,000 * 4 bytes = 14,400,000,000 bytes ~= 13.4 GB

Do you even have that much memory in your machine?

你的机器里有那么多内存吗？

---

---

Note that the issue of stack vs heap doesn't even matter unless you have enough memory to begin with.

请注意，除非您有足够的内存来开始，否则堆栈与堆的问题甚至无关紧要。

---

---

Here's a list of possible problems:

以下是可能出现的问题列表：

• You don't have enough memory.
• If the matrix is declared globally, you'll exceed the maximum size of the binary.
• If the matrix is declared as a local array, then you will blow your stack.
• If you're compiling for 32-bit, you have far exceeded the 2GB/4GB addressing limit.

• 你的内存不够。
• 如果矩阵是全局声明的，您将超过二进制文件的最大大小。
• 如果矩阵被声明为一个本地数组，那么你会炸毁你的堆栈。
• 如果您正在为 32 位编译，那么您已经远远超出了 2GB/4GB 的寻址限制。

回答by Branko Dimitrijevic

Does "60.000" actually mean "60000"? If so, the size of the required memory is 60000 * 60000 * sizeof(float), which is roughly 13.4 GB. A typical 32-bit process is limited to only 2 GB, so it is clear why it doesn't fit.

“60.000”真的意味着“60000”吗？如果是这样，则所需内存的大小为60000 * 60000 * sizeof(float)，大约为 13.4 GB。一个典型的 32 位进程被限制为只有 2 GB，所以很明显为什么它不适合。

On the other hand, I don't see why you shouldn't be able to fit that into a 64-bit process, assuming your machine has enough RAM.

另一方面，我不明白为什么你不能将它放入 64 位进程，假设你的机器有足够的 RAM。

回答by Alex Z

To initialise the 2D array of floats that you want, you will need:

要初始化所需的二维浮点数组，您需要：

60000 * 60000 * 4 bytes = 14400000000 bytes

60000 * 60000 * 4 字节 = 14400000000 字节

Which is approximately 14GB of memory. That's a LOT of memory. To even hold that theoretically, you will need to be running a 64bit machine, not to mention one with quite a bit of RAM installed.

这大约是 14GB 的内存。那是很多记忆。即使在理论上保持这一点，您将需要运行 64 位机器，更不用说安装了相当多的 RAM 的机器。

Furthermore, allocating this much memory is almost never necessary in most situations, are you sure no optimisations could be made here?

此外，在大多数情况下几乎不需要分配这么多内存，您确定这里不能进行优化吗？

EDIT:

编辑：

In light of new information from your comments on other answers: You only have 4GB memory (RAM). Your operating system is hence going to have to page at least 9GB on the Hard Drive, in reality probably more. But you also only have 20GB of Hard Drive space. This is barely enough to page all that data, especially if the disk is fragmented. Finally, (I could be wrong because you haven't stated explicitly) it is quite possible that you're running a 32bit machine. This isn't really capable of handling more than 4GB of memory at a time.

根据您对其他答案的评论中的新信息：您只有 4GB 内存 (RAM)。因此，您的操作系统必须在硬盘驱动器上分页至少 9GB，实际上可能更多。但是您也只有 20GB 的硬盘空间。这几乎不足以对所有数据进行分页，尤其是在磁盘碎片化的情况下。最后，（我可能错了，因为您没有明确说明）您很可能正在运行 32 位机器。这并不能真正一次处理超过 4GB 的内存。

回答by Brian Cain

Allocate the memory at runtime -- consider using a memory mapped file as the backing. Like everyone says, 14 gigs is a lot of memory. But it's not unreasonable to find a computer with 14GB of memory, nor is it unreasonable to page the memory as necessary.

在运行时分配内存——考虑使用内存映射文件作为支持。就像每个人所说的那样，14 场演出是很多内存。但是找一台14GB内存的电脑也不是没有道理，必要时把内存分页也不是没有道理。

With a matrix of this size, you will likely become very curious about memory access performance. Remember to consider the cache grain of your target architecture and if your target has a TLB you may be able to use larger pages to relieve some TLB pressure. Then again, if you don't have enough memory you'll likely care only about how fast your storage I/O is.

使用这种大小的矩阵，您可能会对内存访问性能非常好奇。请记住考虑目标架构的缓存粒度，如果您的目标具有 TLB，您可以使用更大的页面来减轻一些 TLB 压力。再说一次，如果您没有足够的内存，您可能只关心存储 I/O 的速度。

If it's not already obvious, you'll need an architecture that supports a 64-bit address space in order to access this memory directly/conveniently.

如果这还不是很明显，您将需要一个支持 64 位地址空间的架构，以便直接/方便地访问此内存。

回答by David Jones

I had this problem too. I did a workaround where I chopped the array into sections (my biggest allowed array was float A_sub_matrix_20[62944560]). When I declared just one of these in main(), it seems to be put in RAM as I got a runtime exception as soon as main() starts. I was able to declare 20buffers of that size as global variables which works (looks like in global form they are stored on the HDD - when I added A_sub_matrix_20[n] to the watch list in VisualStudio it gave a message "reading from file").

我也有这个问题。我做了一个变通方法，将数组切分成几部分（我允许的最大数组是float A_sub_matrix_20[62944560]）。当我在 main() 中仅声明其中一个时，它似乎被放入 RAM 中，因为我在 main() 启动后立即收到运行时异常。我能够将20个该大小的缓冲区声明为有效的全局变量（看起来它们以全局形式存储在 HDD 上 - 当我将 A_sub_matrix_20[n] 添加到 VisualStudio 的监视列表时，它给出了一条消息“从文件中读取”）。