java 如何检测 OutofMemoryError 的原因？

How to detect the cause of OutofMemoryError?

提问by Jim

I have a complaint that a server application of mine is crashing on high load.
It is a web app running in Tomcat 5.
I see the thread dumps and I see that there is an OutOfMemory error

我抱怨我的服务器应用程序在高负载下崩溃。
它是一个运行在Tomcat 5.
我看到线程转储，我看到有一个 OutOfMemory 错误

1TISIGINFO Dump Event "systhrow" (00040000) Detail
"java/lang/OutOfMemoryError" "Failed to create thread: retVal -1073741830, errno 12" >received 1TIDATETIME Date: 2012/07/17 at 20:03:17 1TIFILENAME >Javacore filename:C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt

1TISIGINFO 转储事件“systhrow”(00040000) 详细信息
“java/lang/OutOfMemoryError”“无法创建线程：retVal -1073741830，errno 12”>收到 1TIDATETIME 日期：2012/07/17 文件名：170:10:00 :C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt

The heap information is the following:

堆信息如下：

Maximum Java heap size : 1500m    
Initial Java heap size : 256m

This is the configuration for initial and Max heap size (32 bit java)

这是初始和最大堆大小的配置（32 位 java）

I also see that there is available free heap space

我还看到有可用的空闲堆空间

1STHEAPFREE    Bytes of Heap Space Free: 2D19F3C0   
1STHEAPALLOC   Bytes of Heap Space Allocated: 5DC00000

This is arround 750MB free space, right?

这是大约 750MB 的可用空间，对吗？

And from the thread method analysis I see that the number of Threads is 695of which 49%is java/lang/Object.wait(Native Method)and 39%is in sun/misc/Unsafe.park(Native Method)
Also I see this NO JAVA STACK 1%not sure what that means.
Also 0 treads deadlocked and 2%is Runnable.

从线程方法分析中我看到线程的数量是695其中49%的java/lang/Object.wait(Native Method)和39%是在sun/misc/Unsafe.park(Native Method)
我也看到这NO JAVA STACK 1%不确定这意味着什么。
也 0 踩死锁并且2%是可运行的。

I am not sure how to interpret this information or how to go on from here to detect the underlying cause.
Any help on this?

我不确定如何解释这些信息或如何从这里继续检测根本原因。
有什么帮助吗？

采纳答案by beny23

According to this post:

根据这篇文章：

There are two possible causes of the java.lang.OutOfMemoryError: Failed to create a thread message:

• There are too many threads running and the system has run out of internal resources to create new threads.
• The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java? stack, and a native stack.

java.lang.OutOfMemoryError: Failed to create a thread 消息有两种可能的原因：

• 运行的线程过多，系统内部资源不足，无法创建新线程。
• 系统已用完用于新线程的本机内存。线程需要用于内部 JVM 结构的本机内存，Java？堆栈和本机堆栈。

So this error may well be completely unrelated to memory, just that too many threads are created...

所以这个错误很可能与内存完全无关，只是创建了太多线程......

EDIT:

编辑：

As you've got 695 threads, you would need 695 times the stack size as memory. Considering this poston thread limits, I suspect that you are trying to create too many threads for the available virtual memory space.

由于您有 695 个线程，因此您需要的堆栈大小是内存的 695 倍。考虑到这篇关于线程限制的帖子，我怀疑您正试图为可用的虚拟内存空间创建太多线程。

回答by Flavio

You should start the JVM with the -XX:+HeapDumpOnOutOfMemoryErrorflag. This will produce a heap dump when the OutOfMemoryErroris generated.

您应该使用该-XX:+HeapDumpOnOutOfMemoryError标志启动 JVM 。这将在生成时产生堆转储OutOfMemoryError。

Then, as @Steve said, you can use a tool like MAT to analyze the dump and see which objects are allocated, and who is keeping references to them. This usually will give you some insight on why your JVM is exhausting its memory.

然后，正如@Steve 所说，您可以使用 MAT 之类的工具来分析转储并查看分配了哪些对象，以及谁在保留对它们的引用。这通常会让您对 JVM 耗尽其内存的原因有所了解。

回答by Ben

I know what you mean, it can be confusing to find somewhere to begin.

我知道你的意思，找个地方开始可能会让人困惑。

Have a look at Eclipse Memory Analyzer(MAT). It will use JHat to dump a memory snapshot of your program into a file, which you can re-open and analyze.

看看Eclipse 内存分析器(MAT)。它将使用 JHat 将程序的内存快照转储到一个文件中，您可以重新打开和分析该文件。

The browser for this file outlines all the objects created by the program very neatly, and you can look into various levels to find if something is suspicious.

此文件的浏览器非常整洁地概述了程序创建的所有对象，您可以查看各个级别以查找是否有可疑内容。

---

---

Appending my comments to answer...

附加我的评论以回答...

Right when your executablewebapp crashes, dump it to MAT. MAT will tell you what object is being created a bunch of times. If it's a custom object, and it often is, it's easy to find. If not, you can see its parent, amputate it, and dribble down from there (sorry for the graphic example, I'm not entirely focused on SO at the moment :).

就在您的可执行web 应用程序崩溃时，将其转储到 MAT。MAT 会多次告诉您正在创建什么对象。如果它是自定义对象，而且经常是这样，则很容易找到。如果没有，你可以看到它的父母，截肢它，然后从那里运球（对不起，图形示例，我目前并不完全专注于 SO :）。

Oh, and I forgot to mention, you can run the program several times under several conditions, and make a dump each time. Then, you can analyze each dump for the trend.

哦，我忘了提到，您可以在多种条件下多次运行该程序，并且每次都进行转储。然后，您可以分析每个转储的趋势。

---

---

But in my case what should I use?I have a web app running in Tomcat

但在我的情况下，我应该使用什么？我有一个在 Tomcat 中运行的 Web 应用程序

Sorry, missed this too. If I'm not mistaken, MAT dumps the JVM process, so as long as the VM is running on your box you can dump its process and see what's going on.

不好意思，也错过了。如果我没记错的话，MAT 会转储 JVM进程，因此只要 VM 在您的机器上运行，您就可以转储其进程并查看发生了什么。

---

---

Another comment mutated into partial solution...

另一个评论变成了部分解决方案......

This is becoming more difficult than it actually is. Seriously, it's pretty easy, after you run MAT once or twice to get the hang of things. Run your app until the thing crashes. Dump it. Change something. Run, crash, dump. Repeat. Then, open the dumps in MAT, and compare what looks suspicious.

这变得比实际更困难。说真的，这很容易，在你运行 MAT 一两次来掌握窍门之后。运行你的应用程序直到它崩溃。倾倒吧。改变些什么。运行，崩溃，转储。重复。然后，打开 MAT 中的转储，并比较看起来可疑的内容。

The trickiest part when I was learning this was finding the process IDto dump - which is still not too mind-numbing.

当我学习这个时，最棘手的部分是找到要转储的进程 ID- 这仍然不是太让人麻木。

回答by JoseK

A similar message on IBM WebSphereshows this line

IBM WebSphere 上的一条类似消息显示了这一行

"Failed to create a thread: retVal"

“创建线程失败：retVal”

as indicative of a native OOMwhich means some thread (of the process) is trying to request a large portion of memory on the heap.

作为本机 OOM 的指示，这意味着（进程的）某些线程正在尝试请求堆上的大部分内存。

The IBM link above has a series of steps - some of which are IBM specific. Have a look.

上面的 IBM 链接有一系列步骤 - 其中一些是 IBM 特定的。看一看。

From a native memory usage perspective:

从本机内存使用的角度来看：

• Maximum Java heap settings
• JDBC driver
• JNI code or Native libraries
• garbage collection of unused classes. Ensurethat -Xnoclassgc is not set.
• Thread Pool settings (fixed size thread pools)
• Too many classloaders etc, but these are not very common.
• Number of classes/classloaders from javacores.

• 最大 Java 堆设置
• JDBC驱动程序
• JNI 代码或本机库
• 未使用类的垃圾收集。确保未设置 -Xnoclassgc。
• 线程池设置（固定大小的线程池）
• 太多的类加载器等，但这些并不常见。
• 来自 javacore 的类/类加载器的数量。

Another thing you could look at is the PermGenSpace - how large is it?

您可以查看的另一件事是 PermGenSpace - 它有多大？

This link http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.htmlsuggests

这个链接http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html建议

Increasing the heap allocation actually exacerbates this problem! It decreases the headroom the compiler, and other native components, have to play with. So the solution to my problem was: 1.reduce the heap allocated to the JVM. 2. remove the memory leaks caused by native objects not being freed in a timely fashion.

增加堆分配实际上加剧了这个问题！它减少了编译器和其他本机组件必须使用的空间。所以我的问题的解决方案是：1.减少分配给JVM的堆。2. 消除原生对象没有被及时释放导致的内存泄漏。

Also have you configured a value in server.xml for maxThreads ? The default is 200 but your app seems to have 695 ?

您是否还在 server.xml 中为 maxThreads 配置了一个值？默认值为 200 但您的应用程序似乎有 695 ？

回答by mon

Fix

使固定

Followed the IBM technote java.lang.OutOfMemoryError while creating new threads, specifically 'ulimit' command to increase the value from the default 1024.

在创建新线程时遵循 IBM 技术说明java.lang.OutOfMemoryError，特别是 'ulimit' 命令以增加默认值 1024 的值。

Symptom

症状

[2/25/15 12:47:34:629 EST] 00000049 SystemErr     R java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at java.lang.Thread.startImpl(Native Method)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at java.lang.Thread.start(Thread.java:936)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.stop(InternalSystemBundle.java:251)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.RuntimeBundleActivator.shutdownEclipse(RuntimeBundleActivator.java:54)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook.run(ServerCollaborator.java:878)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5459)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5585)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook.run(ServerCollaborator.java:850)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$StopAction.alarm(ServerCollaborator.java:809)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1815)

Environment

环境

CentOS 6.6 64 bit IBM WAS 8.5.0.2 64 bit

CentOS 6.6 64 位 IBM WAS 8.5.0.2 64 位

References

参考

• Guidelines for setting ulimits (WebSphere Application Server)

• 设置 ulimit 的准则（WebSphere Application Server）