你能在 c 或 c++ 中分配一个非常大的内存块（> 4GB）吗？
声明：本页面是StackOverFlow热门问题的中英对照翻译，遵循CC BY-SA 4.0协议，如果您需要使用它，必须同样遵循CC BY-SA许可，注明原文地址和作者信息，同时你必须将它归于原作者（不是我）：StackOverFlow
原文地址: http://stackoverflow.com/questions/181050/
Warning: these are provided under cc-by-sa 4.0 license. You are free to use/share it, But you must attribute it to the original authors (not me):
StackOverFlow
Can you allocate a very large single chunk of memory ( > 4GB ) in c or c++?
提问by KPexEA
With very large amounts of ram these days I was wondering, it is possible to allocate a single chunk of memory that is larger than 4GB? Or would I need to allocate a bunch of smaller chunks and handle switching between them?
这些天我想知道有非常大量的内存，是否可以分配大于 4GB 的单个内存块？或者我是否需要分配一堆较小的块并处理它们之间的切换？
Why??? I'm working on processing some openstreetmap xml data and these files are huge. I'm currently streaming them in since I can't load them all in one chunk but I just got curious about the upper limits on malloc or new.
为什么？？？我正在处理一些 openstreetmap xml 数据，这些文件很大。我目前正在流式传输它们，因为我无法将它们全部加载到一个块中，但我只是对 malloc 或 new 的上限感到好奇。
采纳答案by Benoit
Short answer: Not likely
简短回答：不太可能
In order for this to work, you absolutely would haveto use a 64-bit processor. Secondly, it would depend on the Operating System support for allocating more than 4G of RAM to a single process.
为了使其工作，您绝对必须使用 64 位处理器。其次，这将取决于操作系统支持为单个进程分配超过 4G 的 RAM。
In theory, it would be possible, but you would have to read the documentation for the memory allocator. You would also be more susceptible to memory fragmentation issues.
理论上，这是可能的，但您必须阅读内存分配器的文档。您也更容易受到内存碎片问题的影响。
There is good information on Windows memory management.
有关于Windows 内存管理的好信息。
回答by ConcernedOfTunbridgeWells
A Primer on physcal and virtual memory layouts
物理和虚拟内存布局入门
You would need a 64-bit CPU and O/S build and almost certainly enough memory to avoid thrashing your working set. A bit of background:
您将需要 64 位 CPU 和 O/S 构建以及几乎可以肯定的足够内存以避免颠簸您的工作集。一点背景：
A 32 bit machine (by and large) has registers that can store one of 2^32 (4,294,967,296) unique values. This means that a 32-bit pointer can address any one of 2^32 unique memory locations, which is where the magic 4GB limit comes from.
32 位机器（大体上）具有可以存储 2^32 (4,294,967,296) 个唯一值之一的寄存器。这意味着 32 位指针可以寻址 2^32 个唯一内存位置中的任何一个，这就是神奇的 4GB 限制的来源。
Some 32 bit systems such as the SPARCV8 or Xeon have MMU's that pull a trick to allow more physical memory. This allows multiple processes to take up memory totalling more than 4GB in aggregate, but each process is limited to its own 32 bit virtual address space. For a single process looking at a virtual address space, only 2^32 distinct physical locations can be mapped by a 32 bit pointer.
某些 32 位系统（例如 SPARCV8 或 Xeon）具有 MMU，可以使用技巧来允许更多物理内存。这允许多个进程总共占用超过 4GB 的内存，但每个进程仅限于自己的 32 位虚拟地址空间。对于查看虚拟地址空间的单个进程，32 位指针只能映射 2^32 个不同的物理位置。
I won't go into the details but This presentation(warning: powerpoint) describes how this works. Some operating systems have facilities (such as those described Here- thanks to FP above) to manipulate the MMU and swap different physical locations into the virtual address space under user level control.
我不会详细介绍，但是这个演示文稿（警告：powerpoint）描述了它是如何工作的。某些操作系统具有操作 MMU 并将不同物理位置交换到用户级别控制下的虚拟地址空间的工具（例如此处描述的那些- 感谢上面的 FP）。
The operating system and memory mapped I/O will take up some of the virtual address space, so not all of that 4GB is necessarily available to the process. As an example, Windows defaults to taking 2GB of this, but can be set to only take 1GB if the /3G switch is invoked on boot. This means that a single process on a 32 bit architecture of this sort can only build a contiguous data structure of somewhat less than 4GB in memory.
操作系统和内存映射 I/O 将占用一些虚拟地址空间，因此并非所有 4GB 空间都必须可供进程使用。例如，Windows 默认占用 2GB，但如果在启动时调用 /3G 开关，则可以设置为仅占用 1GB。这意味着此类 32 位架构上的单个进程只能在内存中构建略小于 4GB 的连续数据结构。
This means you would have to explicitly use the PAEfacilities on Windows or Equivalent facilities on Linuxto manually swap in the overlays. This is not necessarily that hard, but it will take some time to get working.
这意味着您必须明确使用Windows 上的PAE工具或Linux上的等效工具来手动交换覆盖。这不一定那么难，但需要一些时间才能开始工作。
Alternatively you can get a 64-bit box with lots of memory and these problems more or less go away. A 64 bit architecture with 64 bit pointers can build a contiguous data structure with as many as 2^64 (18,446,744,073,709,551,616) unique addresses, at least in theory. This allows larger contiguous data structures to be built and managed.
或者，您可以获得一个具有大量内存的 64 位机器，这些问题或多或少都会消失。具有 64 位指针的 64 位体系结构可以构建具有多达 2^64 (18,446,744,073,709,551,616) 个唯一地址的连续数据结构，至少在理论上是这样。这允许构建和管理更大的连续数据结构。
回答by Martin Beckett
The advantage of memory mapped files is that you can open a file much bigger than 4Gb (almost infinite on NTFS!) and have multiple <4Gb memory windows into it.
It's much more efficent than opening a file and reading it into memory,on most operating systems it uses the built-in paging support.
内存映射文件的优点是您可以打开一个比 4Gb 大得多的文件（在 NTFS 上几乎是无限的！），并且可以在其中打开多个 <4Gb 内存窗口。
它比打开文件并将其读入内存要高效得多，在大多数操作系统上它使用内置的分页支持。
回答by Rob Walker
This shouldn't be a problem with a 64-bit OS (and a machine that has that much memory).
对于 64 位操作系统（以及具有那么多内存的机器），这应该不是问题。
If malloc can't cope then the OS will certainly provide APIs that allow you to allocate memory directly. Under Windows you can use the VirtualAllocAPI.
如果 malloc 无法应对，那么操作系统肯定会提供允许您直接分配内存的 API。在 Windows 下，您可以使用VirtualAllocAPI。
回答by Steven A. Lowe
it depends on which C compiler you're using, and on what platform (of course) but there's no fundamental reason why you cannot allocate the largest chunk of contiguously available memory - which may be less than you need. And of course you may have to be using a 64-bit system to address than much RAM...
这取决于您使用的是哪个 C 编译器，以及在什么平台上（当然），但是没有根本原因为什么您不能分配最大的连续可用内存块 - 这可能比您需要的要少。当然，您可能必须使用 64 位系统来解决比 RAM 多的问题……
see Mallocfor history and details
有关历史和详细信息，请参阅Malloc
call HeapMaxin alloc.h to get the largest available block size
回答by 1800 INFORMATION
Have you considered using memory mapped files? Since you are loading in really huge files, it would seem that this might be the best way to go.
您是否考虑过使用内存映射文件？由于您正在加载非常大的文件，因此这似乎是最好的方法。
回答by Franci Penov
It depends on whether the OS will give you virtual address space that allows addressing memory above 4GB and whether the compiler supports allocating it using new/malloc.
这取决于操作系统是否会为您提供允许寻址 4GB 以上内存的虚拟地址空间，以及编译器是否支持使用 new/malloc 分配它。
For 32-bit Windows you won't be able to get single chunk bigger than 4GB, as the pointer size is 32-bit, thus limiting your virtual address space to 4GB. (You could use Physical Address Extensionto get more than 4GB memory; however, I believe you have to map that memory into the virtualaddress space of 4GB yourself)
对于 32 位 Windows，您将无法获得大于 4GB 的单个块，因为指针大小是 32 位，因此将您的虚拟地址空间限制为 4GB。（您可以使用物理地址扩展来获得超过 4GB 的内存；但是，我相信您必须自己将该内存映射到 4GB 的虚拟地址空间中）
For 64-bit Windows, the VC++ compiler supports 64-bit pointers with theoretical limit of the virtual address space to 8TB.
对于 64 位 Windows，VC++ 编译器支持 64 位指针，虚拟地址空间的理论限制为 8TB。
I suspect the same applies for Linux/gcc - 32-bit does not allow you, whereas 64-bit allows you.
我怀疑这同样适用于 Linux/gcc - 32 位不允许你，而 64 位允许你。
回答by RandomNickName42
As Rob pointed out, VirtualAlloc for Windows is a good option for this, as is an anonymouse file mapping. However, specifically with respect to your question, the answer to "if C or C++" can allocate, the answer is NO THIS IS NOT SUPPORTED EVEN ON WIN7 RC 64
正如 Rob 指出的那样，Windows 的 VirtualAlloc 是一个不错的选择，匿名文件映射也是如此。但是，特别是关于您的问题，“如果 C 或 C++”的答案可以分配，答案是否定的，即使在 WIN7 RC 64 上也不支持
In the PE/COFF specification for exe files, the field which specifies the HEAP reserve and HEAP commit, is a 32 bit quantity. This is in-line with the physical size limitations of the current heap implmentation in the windows CRT, which is just short of 4GB. So, there is no way to allocate more than 4GB from C/C++ (technicall the OS support facilities of CreateFileMapping and VirtualAlloc/VirtualAllocNuma etc... are not C or C++).
在 exe 文件的 PE/COFF 规范中，指定 HEAP 保留和 HEAP 提交的字段是 32 位数量。这符合 windows CRT 中当前堆实现的物理大小限制，仅不足 4GB。因此，无法从 C/C++ 中分配超过 4GB 的空间（技术上来说，CreateFileMapping 和 VirtualAlloc/VirtualAllocNuma 等操作系统支持设施……不是 C 或 C++）。
Also, BE AWAREthat there are underlying x86 or amd64 ABI construct's known as the page table's. This WILLin effect do what you are concerened about, allocating smaller chunks for your larger request, even though this is happining in kernel memory, there is an effect on the overall system, these tables are finite.
此外，要AWARE有潜在的x86或AMD64 ABI构造的称为页表的。这WILL实际上你是concerened一下，为你的要求较大的分配更小的块，尽管这是在内核内存happining什么，还有就是整个系统的影响，这些表是有限的。
If you are allocating memory in such grandious purportions, you would be well advised to allocate based on the allocation granularity (which VirtualAlloc enforces) and also to identify optional flags's or methods to enable larger pages.
如果您以如此宏大的目的分配内存，则建议您根据分配粒度（VirtualAlloc 强制执行）进行分配，并确定可选标志或方法以启用更大的页面。
4kb pages were the initial page size for the 386, subsaquently the pentium added 4MB. Today, the AMD64(Software Optimization Guide for AMD Family 10h Processors) has a maximum page table entry size of 1GB. This mean's for your case here, let's say you just did 4GB, it would require only 4 unique entries in the kernel's directory to locate\assign and permission your process's memory.
4kb 页面是 386 的初始页面大小，随后奔腾增加了 4MB。今天，AMD64（AMD 家族 10h 处理器软件优化指南）的最大页表条目大小为 1GB。这意味着针对您的情况，假设您刚刚做了 4GB，它只需要内核目录中的 4 个唯一条目来定位\分配和许可您的进程的内存。
Microsoft has also released this manualthat articulates some of the finer points of application memory and it's use for the Vista/2008 platform and newer.
Microsoft 还发布了本手册，其中阐明了应用程序内存的一些更精细的要点，并用于 Vista/2008 平台和更新版本。
内容介绍。4关于内存管理器 4虚拟地址空间。5内核虚拟地址空间的动态分配。5x86 架构的详细信息。664 位架构的详细信息。7x86 架构中的内核模式堆栈跳转。7使用过多的池内存。8安全性：地址空间布局随机化。9ASLR 对图像加载地址的影响。9ASLR 的好处.. 11如何创建基于动态的图像。11输入/输出带宽。11微软 SuperFetch。12页面文件写入。12内存管理器和缓存管理器的协调 13预取式聚类。14大文件管理 15休眠和待机。16高级视频模型 16NUMA 支持 17资源分配。17默认节点和关联。18中断亲和力。19应用程序的 NUMA 感知系统功能。19驱动程序的 NUMA 感知系统功能。19分页。20可扩展性。20效率和并行性.. 20页框编号和 PFN 数据库。20大页面。21缓存对齐的池分配。21虚拟机。22负载均衡。22其他优化。23系统完整性。23硬件错误诊断。23代码完整性和驱动程序签名。24错误检查期间的数据保留。24你应该做什么。24对于硬件制造商。24对于驱动程序开发人员。24对于应用程序开发人员。25对于系统管理员。25资源。25回答by Dan Olson
If size_t is greater than 32 bits on your system, you've cleared the first hurdle. But the C and C++ standards aren't responsible for determining whether any particular call to new or malloc succeeds (except malloc with a 0 size). That depends entirely on the OS and the current state of the heap.
如果您系统上的 size_t 大于 32 位，则您已经清除了第一个障碍。但是 C 和 C++ 标准不负责确定对 new 或 malloc 的任何特定调用是否成功（大小为 0 的 malloc 除外）。这完全取决于操作系统和堆的当前状态。
回答by Zachary Hamm
Like everyone else said, getting a 64bit machine is the way to go. But even on a 32bit machine intel machine, you can address bigger than 4gb areas of memory if your OS and your CPU support PAE. Unfortunately, 32bit WinXP does not do this (does 32bit Vista?). Linux lets you do this by default, but you will be limited to 4gb areas, even with mmap() since pointers are still 32bit.
就像其他人所说的那样，获得 64 位机器是必经之路。但即使在 32 位机器英特尔机器上，如果您的操作系统和 CPU 支持PAE，您也可以寻址大于 4GB 的内存区域。不幸的是，32 位 WinXP 不这样做（32 位 Vista 是吗？）。Linux 默认允许您执行此操作，但您将被限制为 4gb 区域，即使使用 mmap()，因为指针仍然是 32 位。
What you should do though, is let the operating system take care of the memory management for you. Get in an environment that can handle that much RAM, then read the XML file(s) into (a) data structure(s), and let it allocate the space for you. Then operate on the data structure in memory, instead of operating on the XML file itself.
不过，您应该做的是让操作系统为您处理内存管理。进入一个可以处理那么多 RAM 的环境，然后将 XML 文件读入 (a) 数据结构，并让它为您分配空间。然后对内存中的数据结构进行操作，而不是对XML文件本身进行操作。
Even in 64bit systems though, you're not going to have a lot of control over what portions of your program actually sit in RAM, in Cache, or are paged to disk, at least in most instances, since the OS and the MMU handle this themselves.
即使在 64 位系统中，您也不会对程序的哪些部分实际位于 RAM、缓存中或分页到磁盘中进行大量控制，至少在大多数情况下，因为操作系统和 MMU 处理这自己。
