Linux Java 拒绝启动 - 无法为对象堆保留足够的空间

Java Refuses to Start - Could not reserve enough space for object heap

提问by Randyaa

Background

背景

We have a pool of aproximately 20 linux blades. Some are running Suse, some are running Redhat. ALL share NAS space which contains the following 3 folders:

我们有大约 20 个 linux 刀片池。有些运行 Suse，有些运行 Redhat。ALL 共享 NAS 空间，其中包含以下 3 个文件夹：

• /NAS/app/java - a symlink that points to an installation of a Java JDK. Currently version 1.5.0_10
• /NAS/app/lib - a symlink that points to a version of our application.
• /NAS/data - directory where our output is written

• /NAS/app/java - 指向 Java JDK 安装的符号链接。当前版本 1.5.0_10
• /NAS/app/lib - 指向我们应用程序版本的符号链接。
• /NAS/data - 写入输出的目录

All our machines have 2 processors (hyperthreaded) with 4gb of physical memory and 4gb of swap space. We limit the number of 'jobs' each machine can process at a given time to 6 (this number likely needs to change, but that does not enter into the current problem so please ignore it for the time being).

我们所有的机器都有 2 个处理器（超线程），4gb 物理内存和 4gb 交换空间。我们将每台机器在给定时间可以处理的“作业”数量限制为 6（这个数字可能需要更改，但这不会影响当前问题，因此请暂时忽略它）。

Some of our jobs set a Max Heap size of 512mb, some others reserve a Max Heap size of 2048mb. Again, we realize we could go over our available memory if 6 jobs started on the same machine with the heap size set to 2048, but to our knowledge this has not yet occurred.

我们的一些作业设置了 512mb 的最大堆大小，而其他一些则保留了 2048mb 的最大堆大小。同样，我们意识到如果在同一台机器上启动 6 个作业并将堆大小设置为 2048，我们可以检查可用内存，但据我们所知，这还没有发生。

The Problem

问题

Once and a while a Job will fail immediately with the following message:

有时，作业会立即失败并显示以下消息：

Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

We used to chalk this up to too many jobs running at the same time on the same machine. The problem happened infrequently enough (MAYBEonce a month) that we'd just restart it and everything would be fine.

我们过去常常将其归结为在同一台机器上同时运行太多作业。这个问题很少发生（可能一个月一次），我们只需重新启动它，一切都会好起来的。

The problem has recently gotten much worse. All of our jobs which request a max heap size of 2048m fail immediately almost every time and need to get restarted several times before completing.

这个问题最近变得更糟了。我们所有要求最大堆大小为 2048m 的作业几乎每次都会立即失败，并且在完成之前需要重新启动几次。

We've gone out to individual machines and tried executing them manually with the same result.

我们已经在单独的机器上尝试手动执行它们并得到相同的结果。

Debugging

调试

It turns out that the problem only exists for our SuSE boxes. The reason it has been happening more frequently is becuase we've been adding more machines, and the new ones are SuSE.

事实证明，这个问题只存在于我们的 SuSE 盒子上。它发生得更频繁的原因是因为我们一直在添加更多的机器，而新的机器是 SuSE。

'cat /proc/version' on the SuSE boxes give us:

SuSE 盒子上的“cat /proc/version”给我们：

Linux version 2.6.5-7.244-bigsmp (geeko@buildhost) (gcc version 3.3.3 (SuSE Linux)) #1 SMP Mon Dec 12 18:32:25 UTC 2005

'cat /proc/version' on the RedHat boxes give us:

RedHat 盒子上的“cat /proc/version”给了我们：

Linux version 2.4.21-32.0.1.ELsmp ([email protected]) (gcc version 3.2.3 20030502 (Red Hat Linux 3.2.3-52)) #1 SMP Tue May 17 17:52:23 EDT 2005

'uname -a' gives us the following on BOTH types of machines:

'uname -a' 在两种类型的机器上为我们提供以下信息：

UTC 2005 i686 i686 i386 GNU/Linux

No jobs are running on the machine, and no other processes are utilizing much memory. All of the processes currently running mightbe using 100mb total.

机器上没有正在运行的作业，也没有其他进程正在使用大量内存。当前运行的所有进程可能总共使用 100mb。

'top' currently shows the following:

'top' 目前显示以下内容：

Mem:   4146528k total,  3536360k used,   610168k free,   132136k buffers
Swap:  4194288k total,        0k used,  4194288k free,  3283908k cached

'vmstat' currently shows the following:

“vmstat”当前显示以下内容：

procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r  b   swpd   free   buff  cache   si   so    bi    bo   in    cs us sy id wa
0  0      0 610292 132136 3283908    0    0     0     2   26    15  0  0 100  0

If we kick off a job with the following command line (Max Heap of 1850mb) it starts fine:

如果我们使用以下命令行（最大堆为 1850mb）启动一项工作，它开始正常：

java/bin/java -Xmx1850M -cp helloworld.jar HelloWorld
Hello World

If we bump up the max heap size to 1875mb it fails:

如果我们将最大堆大小提高到 1875mb，它就会失败：

java/bin/java -Xmx1875M -cp helloworld.jar HelloWorld
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

It's quite clear that the memory currently being used is for Buffering/Caching and that's why so little is being displayed as 'free'. What isn't clear is why there is a magical 1850mb line where anything higher means Java can't start.

很明显，当前使用的内存用于缓冲/缓存，这就是为什么显示为“空闲”的内存如此之少。不清楚的是为什么有一个神奇的 1850mb 行，其中任何更高的行都意味着 Java 无法启动。

Any explanations would be greatly appreciated.

任何解释将不胜感激。

回答by Johan Guiheneuf

What is the JVM used ? I know that BEA JRockit does not exceed 1850mB for max heap size. It does not fail but warns the user that it will not use more than 1850mB.

使用的 JVM 是什么？我知道 BEA JRockit 的最大堆大小不超过 1850mB。它不会失败，但会警告用户它不会使用超过 1850mB。

I don't know why there is such a limit but I know it exists for BEA JRockit.

我不知道为什么会有这样的限制，但我知道 BEA JRockit 存在这种限制。

Best regards.

此致。

回答by Gareth Davis

This maybe way off the track, but there are two things that spring to mind. Both of the following assume that you are running a 32bit version of Linux.

这可能偏离了轨道，但有两件事会浮现在脑海中。以下两个都假设您运行的是 32 位版本的 Linux。

There is a process size limit on linux, seem to remember on CentOS was around 2.5gb and is configured in the kernel (i.e. a recomple to change). Your process might be hitting that once you add up all the JVM code + Permgen space and all the rest of the JVM libraries.

linux 上有一个进程大小限制，似乎记得在 CentOS 上是 2.5gb 左右，并且是在内核中配置的（即重新编译更改）。一旦您将所有 JVM 代码 + Permgen 空间和所有其余 JVM 库相加，您的进程可能会遇到这种情况。

The second thing is something I've come across that you may be running out of address space, sounds wierd I know. Had a problem running Glassfish with a 1.5Gb heap, when it tried to complie a JSP my forking javac it would fail because the OS couldn't allocate enough address space for the newly created process even though there was 12gb of memory in the box. There may be something similar going on here.

第二件事是我遇到的事情，你可能会用完地址空间，我知道这听起来很奇怪。运行带有 1.5Gb 堆的 Glassfish 时出现问题，当它尝试编译 JSP 时，我的分支 javac 会失败，因为操作系统无法为新创建的进程分配足够的地址空间，即使盒子中有 12GB 的内存。这里可能有类似的事情发生。

I'm afraid the only solutions to the above two where to upgrade to a 64bit kernel.

恐怕只有以上两种解决方案才能升级到64位内核。

Hope this is of some use.

希望这有点用。

回答by Gareth Davis

It seems that for 32-bit servers there is a JVM limitation that cannot be overcome (unless you find a special 32-bit JVM that does not impose a 2GB limit or less).

似乎对于 32 位服务器有一个无法克服的 JVM 限制（除非您找到一个特殊的 32 位 JVM，它不会强加 2GB 或更少的限制）。

This thread on The Server Side has more details including several people who tested out various JVMs on 32-bit architectures. IBM's JVM seems to allow 100 more MB but that's not really going to get you what you want.

The Server Side 上的这个帖子有更多细节，包括几个在 32 位架构上测试各种 JVM 的人。IBM 的 JVM 似乎允许再增加 100 MB，但这并不能真正满足您的需求。

http://www.theserverside.com/discussions/thread.tss?thread_id=26347

http://www.theserverside.com/discussions/thread.tss?thread_id=26347

The "real" solution is to use a 64-bit server with a 64-bit JVM to get heaps larger than 2GB per process. However, it's important to also consider the impact of increasing your address size (not just the addressable space) by using a 64-bit JVM. There will likely be performance and memory impacts for processing using less than 4GB of memory.

“真正的”解决方案是使用带有 64 位 JVM 的 64 位服务器来获得每个进程大于 2GB 的堆。但是，还必须考虑使用 64 位 JVM 增加地址大小（不仅仅是可寻址空间）的影响。使用少于 4GB 的内存进行处理可能会影响性能和内存。

Food for thought: do each of these jobs really require 2GB of memory? Is there any way for the jobs to be modified to run within 1.8GB so this limit is not a problem?

深思熟虑：这些工作中的每一个都真的需要 2GB 的内存吗？有什么办法可以修改作业在 1.8GB 内运行，所以这个限制不是问题吗？

回答by Jon Bright

You're using a 32-bit OS, so you're going to be seeing limits on the total size due to that. Other answers have covered this in more detail, so I'll avoid repeating their information.

您使用的是 32 位操作系统，因此您将看到总大小受到限制。其他答案更详细地涵盖了这一点，因此我将避免重复他们的信息。

A behaviour that I noticed with our servers recently is that specifying a maximum heap size with -Xmxwhile not specifying a minimum heap size with -Xmswould lead to Java's server VM immediately attempting to allocate all of the memory needed for the maximum heap size. And sure, if the app gets up to that heap size, that's the amount of memory that you'll need. But the chances are, your apps will be starting out with comparitively small heaps and mayrequire the larger heap at some later point. Additionally specifying the minimum heap size will let you start your app start with a smaller heap and gradually grow that heap.

我最近在我们的服务器上注意到的一个行为是指定最大堆大小-Xmx而不指定最小堆大小-Xms会导致 Java 的服务器 VM 立即尝试分配最大堆大小所需的所有内存。当然，如果应用程序达到该堆大小，这就是您需要的内存量。但很有可能，您的应用程序开始时使用的堆相对较小，稍后可能需要更大的堆。此外，指定最小堆大小将使您的应用程序从较小的堆开始，然后逐渐增大该堆。

All of this isn't going to help you increase your maximum heap size, but I figured it might help, so...

所有这些都不会帮助您增加最大堆大小，但我认为它可能会有所帮助，所以......

回答by Andrew

ulimit max memory size and virtual memory set to unlimited?

ulimit 最大内存大小和虚拟内存设置为无限制？

回答by Hyman Leow

Given that none of the other suggestions have worked (including many things I'd have suggested myself), to help troubleshoot further, could you try running:

鉴于其他建议均无效（包括我自己建议的许多内容），为了进一步排除故障，您可以尝试运行：

sysctl -a

On both the SuSE and RedHat machines to see if there are any differences? I'm guessing the default configurations are different between these two distributions that's causing this.

在SuSE和RedHat机器上看看有没有什么区别？我猜这两个发行版之间的默认配置是不同的。

回答by MarkR

Running a 32-bit OS is a mistake; you should definitely upgrade at the earliest convenience.

运行 32 位操作系统是错误的；你一定要尽早升级。

I don't know whether Java requires its heap to be in a single contiguous chunk, but if it does, asking for 1.8G of heap on a 32-bit box sounds like a tall order. You're assuming that there is a chunk of address space, almost half of it, free at JVM startup time.

我不知道 Java 是否要求它的堆位于一个连续的块中，但如果是这样，那么在 32 位机器上要求 1.8G 的堆听起来是一项艰巨的任务。您假设有一大块地址空间，几乎有一半在 JVM 启动时空闲。

Depending on what other libraries are loaded at the time, there may not be. Libraries can allocate memory anywhere they like, so it could fragment your address space sufficiently that 1.8G is not available in one chunk.

根据当时加载的其他库，可能没有。库可以在他们喜欢的任何地方分配内存，因此它可以充分分割您的地址空间，以至于 1.8G 在一个块中不可用。

There is only about 3G address space max available on Linux 32-bit anyway. Libraries and the JVM itself uses some to start with.

无论如何，Linux 32 位上最多只有大约 3G 地址空间可用。库和 JVM 本身使用一些开始。

回答by Luca Tettamanti

As suggested in other responses the problem is causes by exhaustion of virtual address space. A 32bit linux userspace program is usually limited to 3GB of AS; the remaining 1GB is used by the kernel (rationale: since the top 1GB is kernel fixed mapping it's not necessary to touch the page table when serving syscalls).

正如其他响应中所建议的，该问题是由虚拟地址空间耗尽引起的。一个 32 位的 linux 用户空间程序通常限制为 3GB 的 AS；剩余的 1GB 由内核使用（理由：由于前 1GB 是内核固定映射，因此在服务系统调用时没有必要接触页表）。

RHEL kernels, however, implement the so called 4GB/4GB split where the full 4GB AS is available to userspace processes at cost of a slight runtime overhead (the kernel lives in a separate 4GB virtual AS)

然而，RHEL 内核实现了所谓的 4GB/4GB 拆分，其中完整的 4GB AS 可用于用户空间进程，但运行时开销很小（内核位于单独的 4GB 虚拟 AS 中）

回答by Peter Lawrey

You need to look at upgrading your OS and Java. Java 5.0 is EOL but if you cannot update to Java 6, you could use the latest patch level 22!

您需要考虑升级您的操作系统和 Java。Java 5.0 已停产，但如果您无法更新到 Java 6，则可以使用最新的补丁级别 22！

32-bit Windows is limited to ~ 1.3 GB so you are doing well to se the maximum to 1.8. Note: this is a problem with continous memory, and as your system runs its memory space can get fragmented so it does not suprise me you have this problem.

32 位 Windows 限制为 ~ 1.3 GB，因此您可以很好地将最大值设置为 1.8。注意：这是连续内存的问题，当你的系统运行时，它的内存空间可能会碎片化，所以我不会惊讶你有这个问题。

A 64-bit OS, doesn't have this problem as it has much more virtual space, you don't even have to upgrade to a 64-bit version of java to take advantage of this.

64 位操作系统没有这个问题，因为它有更多的虚拟空间，你甚至不必升级到 64 位版本的 java 来利用这一点。

BTW, in my experience, 32-bit Java 5.0 can be faster than 64-bit Java 5.0. It wasn't until many years later that Java 6 update 10 was faster for 64-bit.

顺便说一句，根据我的经验，32 位 Java 5.0 可以比 64 位 Java 5.0 快。直到多年之后，Java 6 update 10 才在 64 位上变得更快。

回答by Penz

I upgraded the memory of a machine from 2GB to 4GB, and started to get the error straight away:

我将一台机器的内存从 2GB 升级到 4GB，并立即开始出现错误：

$ java -version
Error occurred during initialization of VM
Could not reserve enough space for object heap
Could not create the Java virtual machine.

The problem was the ulimit, which I had set at 1GB for the addressable space. Increasing it to 2GB solved the issue.

问题是 ulimit，我为可寻址空间设置了 1GB。将其增加到 2GB 解决了这个问题。

-Xms and -Xmx had no effect.

-Xms 和 -Xmx 没有效果。

Looks like java tries to get memory in proportion to the available memory, and fails if it can't.

看起来 java 试图获取与可用内存成比例的内存，如果不能，则失败。