windows 64 位大型 malloc

64 bit large mallocs

提问by SPWorley

What are the reasons a malloc() would fail, especially in 64 bit?

malloc() 失败的原因是什么，尤其是在 64 位中？

My specific problem is trying to malloc a huge 10GB chunk of RAM on a 64 bit system. The machine has 12GB of RAM, and 32 GB of swap. Yes, the malloc is extreme, but why would it be a problem? This is in Windows XP64 with both Intel and MSFT compilers. The malloc sometimes succeeds, sometimes doesn't, about 50%. 8GB mallocs always work, 20GB mallocs always fail. If a malloc fails, repeated requests won't work, unless I quit the process and start a fresh process again (which will then have the 50% shot at success). No other big apps are running. It happens even immediately after a fresh reboot.

我的具体问题是试图在 64 位系统上分配一个巨大的 10GB 内存块。这台机器有 12GB 的 RAM 和 32GB 的交换空间。是的，malloc 是极端的，但为什么会出现问题呢？这是在带有 Intel 和 MSFT 编译器的 Windows XP64 中。malloc 有时成功，有时不成功，大约 50%。8GB mallocs 总是有效，20GB mallocs 总是失败。如果 malloc 失败，重复请求将不起作用，除非我退出进程并再次启动一个新进程（这将有 50% 的成功率）。没有其他大型应用程序正在运行。它甚至在重新启动后立即发生。

I could imagine a malloc failing in 32 bit if you have used up the 32 (or 31) bits of address space available, such that there's no address range large enough to assign to your request.

如果您已经用完了 32（或 31）位可用地址空间，我可以想象 malloc 在 32 位失败，这样就没有足够大的地址范围来分配给您的请求。

I could also imagine malloc failing if you have used up your physical RAM andyour hard drive swap space. This isn't the case for me.

如果您的物理 RAM和硬盘驱动器交换空间用完，我也可以想象 malloc 会失败。这不是我的情况。

But why else could a malloc fail? I can't think of other reasons.

但是为什么 malloc 会失败呢？我想不出其他原因。

I'm more interested in the general malloc question than my specific example, which I'll likely replace with memory mapped files anyway. The failed malloc() is just more of a puzzle than anything else... that desire to understand your tools and not be surprised by the fundamentals.

我对一般的 malloc 问题比我的具体示例更感兴趣，无论如何我可能会用内存映射文件替换它。失败的 malloc() 比其他任何事情都更像是一个难题……希望了解您的工具而不是对基础知识感到惊讶。

回答by workmad3

malloc tries to allocate a contiguous memory range, and this will initially be in real memory simply due to how swap memory works (at least as far as I remember). It could easily be that your OS sometimes can't find a contiguous block of 10gb of memory and still leave all the processes that require real memory in RAM at the same time (at which point your malloc will fail).

malloc 尝试分配一个连续的内存范围，这最初将在实际内存中，这仅仅是由于交换内存的工作方式（至少据我所知）。很可能您的操作系统有时无法找到连续的 10GB 内存块，并且仍然将所有需要实际内存的进程同时留在 RAM 中（此时您的 malloc 将失败）。

Do you actually require 10gb of contiguous memory, or would you be able to wrap a storage class/struct around several smaller blocks and use your memory in chunks instead? This relaxes the huge contiguous requirement and should also allow your program to use the swap file for less used chunks.

您实际上是否需要 10gb 的连续内存，或者您是否能够将存储类/结构包装在几个较小的块周围并以块的形式使用您的内存？这放宽了巨大的连续要求，并且还应该允许您的程序将交换文件用于较少使用的块。

回答by bk1e

Have you tried using VirtualAlloc()and VirtualFree()directly? This may help isolate the problem.

您是否尝试过直接使用VirtualAlloc()和VirtualFree()？这可能有助于隔离问题。

• You'll be bypassing the C runtime heap and the NT heap.
• You can reserve virtual address space and then commit it. This will tell you which operation fails.

• 您将绕过 C 运行时堆和 NT 堆。
• 您可以保留虚拟地址空间，然后提交。这将告诉您哪个操作失败。

If the virtual address space reservation fails (even though it shouldn't, judging from what you've said), Sysinternals VMMapmay help explain why. Turn on "Show free regions" to look at how the free virtual address space is fragmented.

如果虚拟地址空间预留失败（即使它不应该，从你所说的来看），Sysinternals VMMap可能有助于解释原因。打开“显示空闲区域”以查看空闲虚拟地址空间是如何碎片化的。

回答by SmacL

Just a guess here, but malloc allocates contiguous memory and you may not have a sufficiently large contiguous section on your heap. Here's a few things I would try;

这里只是一个猜测，但 malloc 分配连续内存，您的堆上可能没有足够大的连续部分。这是我会尝试的一些事情；

Where a 20GB malloc fails, do four 5GB mallocs succeed? If so, it is a contiguous space issue.

如果 20GB malloc 失败，四个 5GB malloc 会成功吗？如果是这样，这是一个连续的空间问题。

Have you checked your compiler switches for anything that limits total heap size, or largest heap block size?

您是否检查过编译器开关是否有任何限制总堆大小或最大堆块大小的内容？

Have you tried writing a program that declares a static variable of the required size? If this works you could implement your own heap with big mallocs in that space.

您是否尝试编写声明所需大小的静态变量的程序？如果这有效，您可以在该空间中使用大型 malloc 实现您自己的堆。

回答by bdonlan

Have you tried using heap functionsto allocate your memory instead?

您是否尝试过使用堆函数来分配内存？

回答by RandomNickName42

Here's an official source that states the maximum request size of the heap is defined by your linked CRT library (aside from your previous code having integer overflows going to 0 which is why you didn't get NULL back) (_HEAP_MAXREQ).

这是一个官方来源，说明堆的最大请求大小由您链接的 CRT 库定义（除了您之前的代码具有整数溢出到 0 这就是为什么您没有返回 NULL）（_HEAP_MAXREQ）。

http://msdn.microsoft.com/en-us/library/6ewkz86d.aspx

http://msdn.microsoft.com/en-us/library/6ewkz86d.aspx

Check out my answer herefor large windows allocations, I include a reference to a MS paper on Vista/2008 memory model advancements.

在此处查看我对大 Windows 分配的回答，我包括对有关 Vista/2008 内存模型改进的 MS 论文的参考。

In short, the stock CRT does not support, even for a native 64 bit process any heap size larger than 4gb. You haveto use VirtualAlloc* or CreateFileMapping or some other analogues.

简而言之，即使对于本机 64 位进程，库存 CRT 也不支持任何大于 4gb 的堆大小。您必须使用 VirtualAlloc* 或 CreateFileMapping 或其他一些类似物。

Oh I also noticed you are claiming that your larger allocations are actually succeeding, this is actually incorrect, you are mis-interpreting the malloc(0x200000000); (that's 8gb in hex), what is happening is you are requesting a 0 byte allocation due to a cast or some other effect of your test harness, you are most definitely not observing any thing larger than a 0xfffff000 bytes heap being committed, it is simply you are seeing integer overflows down casting.

哦，我还注意到您声称您的较大分配实际上是成功的，这实际上是不正确的，您误解了 malloc(0x200000000); （这是 8gb 的十六进制），正在发生的事情是由于强制转换或测试工具的其他影响，您正在请求 0 字节分配，您绝对没有观察到任何大于 0xfffff000 字节堆的东西被提交，它是只是你看到整数溢出向下转换。

WORD TO THE WYSEor * TIPS TO SAVE YOUR HEAP SANITY*

WYSE 的话或 * 保存您的 HEAP SANITY 的提示*

THE ONLYWAY TO ALLOCATE MEMORY WITH MALLOC (OR ANY OTHER DYNAMIC REQUEST)

THE ONLY的方式来分配内存使用malloc（或任何其它动态请求）

void *foo = malloc(SIZE);

THE VALUE OF A DYNAMIC MEMORY REQUEST MUST NEVER (I CAN NOT STRESS THAT ENOUGH) BE CALCULATED WITHIN THE "()" PAREN'S OF THE REQUEST

动态内存请求的值绝不能（我不能强调足够）在请求的“()”父级内计算

mytype *foo = (mytype *) malloc(sizeof(mytype) * 2);

The danger is that an integeroverflow will occur.

危险是会发生整数溢出。

It is always a coding ERRORto perform arithmetic at the time of the call, you MUST ALWAYScalculate the TOTAL SUMof data to be requested before the statement which evaluates the request.

它始终是一个编码错误在调用的时间来执行算术，你必须一直计算总和，其计算的请求的语句之前请求的数据。

Why is it so bad? We know this is a mistake, because the point at which a requestis made for dynamic resources, there mustbe a point in the future where we will usethis resource.

为什么这么糟糕？我们知道这是一个错误，因为在请求动态资源的那个点，未来肯定会有一个点我们会使用这个资源。

To use what we have requested we mustknow how large it is ? (e.g. the array count, the type size, etc..).

要使用我们所要求的，我们必须知道它有多大？（例如数组计数、类型大小等）。

This would mean, if we ever see any arithmetic at all, inside the () of a resource request, it is an error as we MUSTduplicate that code againin order to use that data appropriately.

这意味着，如果我们在资源请求的 () 中看到任何算术，那将是一个错误，因为我们必须再次复制该代码才能正确使用该数据。

回答by user2313952

The problem is that Visual Studio does not define WIN64 when you compile a 64 bit application, it usually still keeps WIN32, which is wrong for 64 bit apps. This then causes the run-time to use the 32-bit value when _HEAP_MAXREQis defined, so all large malloc()will fail. If you change your project (under project properties, preprocessed definitions) to be WIN64, then the very large malloc()should have no trouble at all.

问题是Visual Studio在编译64位应用程序时并没有定义WIN64，它通常仍然保持WIN32，这对于64位应用程序来说是错误的。这会导致运行时_HEAP_MAXREQ在定义时使用 32 位值，因此所有大的malloc()都将失败。如果您将项目（在项目属性下，预处理定义下）更改为 WIN64，那么非常大malloc()应该完全没有问题。

回答by Tim Lovell-Smith

I found the question interesting so I tried to research it, from a theoretical POV:

我发现这个问题很有趣，所以我尝试从理论 POV 研究它：

In 64-bit (actually 48-bit usable due to chip limitations, and less (44 bits?) due to OS limitations) you should certainly should notbe limited by virtual memory fragmentation, i.e. lack of contiguous virtual address space. The reason is there is just so much virtual address space that it is quite impractical to exhaust it.

在 64 位（由于芯片限制实际上 48 位可用，由于操作系统限制而更少（44 位？））你当然不应该受到虚拟内存碎片的限制，即缺乏连续的虚拟地址空间。原因是虚拟地址空间太大，用完它是不切实际的。

Also, we can expect that physicalmemory fragmentation should not be an issue, as virtual memory means there doesn't need to be a contiguous physical memory address range in order to satisfy an allocation request. Instead it can be satisfied with any sufficiently large set of memory pages.

此外，我们可以预期物理内存碎片应该不是问题，因为虚拟内存意味着不需要连续的物理内存地址范围来满足分配请求。相反，它可以满足任何足够大的内存页面集。

So you must be running into something else: o.e. some other limitation that applies to virtual memory.

因此，您必须遇到其他问题： oe 其他一些适用于虚拟内存的限制。

One other limit which definitely exists on Windows is the commit limit. More information on this:

Windows 上肯定存在的另一个限制是提交限制。更多信息：

http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

http://blogs.technet.com/b/markrussinovich/archive/2008/11/17/3155406.aspx

Other possible limits could exist, e.g. quirk of how the actual implementation has to work with the actual hardware. Imagine that when trying to create a mapping of virtual address space to physical address space you run out of entries in the page table to do the virtual address mapping... does the OS memory allocator code care to handle this unlikely scenario? Perhaps not...

可能存在其他可能的限制，例如实际实现必须如何与实际硬件一起工作的怪癖。想象一下，当尝试创建虚拟地址空间到物理地址空间的映射时，您用尽了页表中的条目来进行虚拟地址映射……操作系统内存分配器代码是否关心处理这种不太可能的情况？也许不是...

You can read more information on how page tables actually work to do virtual address translation here:

您可以在此处阅读有关页表如何实际工作以进行虚拟地址转换的更多信息：

http://en.wikipedia.org/wiki/Memory_management_unit

http://en.wikipedia.org/wiki/Memory_management_unit

回答by dmityugov

But why else could a malloc fail? I can't think of other reasons

但是为什么 malloc 会失败呢？我想不出其他原因

As implicitly stated previously several times, because of memory fragmentation

正如前面多次暗示的那样，因为内存碎片

回答by sybreon

It is most likely fragmentation. For simplicity, let's use an example.

这很可能是碎片化。为简单起见，让我们举一个例子。

The memory consists of a single 12kb module. This memory is organised into 1kb blocks in the MMU. So, you have 12 x 1kb blocks. Your OS uses 100 bytes but this is basically the code that manages the page tables. So, you cannot swap it out. Then, your apps all use 100 bytes each.

内存由单个 12kb 模块组成。该内存在 MMU 中被组织成 1kb 的块。所以，你有 12 x 1kb 块。您的操作系统使用 100 字节，但这基本上是管理页表的代码。所以，你不能换掉它。然后，您的应用程序每个都使用 100 个字节。

Now, with just your OS and your application running (200 bytes), you would already be using 200 bytes of memory (occupying 2kb blocks). Leaving exactly 10kb available for malloc().

现在，只要您的操作系统和应用程序在运行（200 字节），您就已经使用了 200 字节的内存（占用 2kb 块）。只剩下 10kb 可用于malloc().

Now, you started by malloc()a couple of buffers - A (900 byte), B (200 byte). Then, you free up A. Now, you have 9.8kb free (non-contiguous). So, you try to malloc()C (9kb). Suddenly, you fail.

现在，您从malloc()几个缓冲区开始- A（900 字节）、B（200 字节）。然后，您释放了 A。现在，您有 9.8kb 的可用空间（非连续）。所以，你试试malloc()C（9kb）。突然，你失败了。

You have 8.9k contiguous at the tail end and 0.9k at the front end. You cannot re-map the first block to the end because B stretches over the first 1k and the second 1k blocks.

尾部有 8.9k 连续，前端有 0.9k。您不能将第一个块重新映射到末尾，因为 B 延伸到第一个 1k 块和第二个 1k 块。

You can still malloc()a single 8kb block.

您仍然可以malloc()使用单个 8kb 块。

Granted, this example is a little contrived, but hope it helps.

当然，这个例子有点做作，但希望它有所帮助。