Tomcat 崩溃：Java 运行时环境的内存不足，无法继续

Tomcat crash: There is insufficient memory for the Java Runtime Environment to continue

提问by user2792796

We are running a webapplication with low number of users, about 7, but with high load of showing images. Recently we upgraded to Java 6u43 and Tomcat 6.036 and after some usage (about a week) tomcat services crashes. After a restart it can run for a few days, or just a few hours and crashes again. Even after reboot of the server it crashes again, we even saw that after 10 minutes of being active.

我们正在运行一个用户数量较少的 Web 应用程序，大约 7 个，但显示图像的负载很高。最近我们升级到 Java 6u43 和 Tomcat 6.036，在使用一段时间后（大约一周），tomcat 服务崩溃了。重新启动后，它可以运行几天，或者几个小时，然后再次崩溃。即使在重新启动服务器后它再次崩溃，我们甚至在活动 10 分钟后看到了。

it is always with the same message in a hs_err_pid file in the bin folder of tomcat. See below an example of the full error message. It is always insufficient native memory and the main active thread is always JavaThread "C2 CompilerThread1", but always in a different method.

在tomcat的bin文件夹中的hs_err_pid文件中总是带有相同的消息。请参阅下面的完整错误消息示例。它总是本机内存不足，主活动线程总是 JavaThread "C2 CompilerThread1"，但总是在不同的方法中。

We are running on windows 2008 service pack 2, 32 bit, 2 CPU, 3GB. For tomcat xms set to 512KB and xmx to 1536KB. no other specific settings. When it crashes the Java heap is far below full usage of the 1536.

我们在 Windows 2008 Service Pack 2、32 位、2 个 CPU、3GB 上运行。对于 tomcat xms 设置为 512KB 和 xmx 为 1536KB。没有其他特定设置。当它崩溃时，Java 堆远远低于 1536 的完全使用率。

Any idea, suggestions what this could be?

任何想法，建议这可能是什么？

(before the tomcat upgrade we used tomcat 6.014 and Java 6u02 for long time without any issue)

（在tomcat升级之前我们使用tomcat 6.014和Java 6u02很长时间没有任何问题）

#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 629784 bytes for Chunk::new
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.cpp:317), pid=5156, tid=1820
#
# JRE version: 6.0_43-b01
# Java VM: Java HotSpot(TM) Server VM (20.14-b01 mixed mode windows-x86 )

---------------  T H R E A D  ---------------

Current thread (0x00f99800):  JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)]

Stack: [0x66680000,0x666f0000],  sp=0x666ee694,  free space=441k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)

[error occurred during error reporting (printing native stack), id 0xc0000005]


Current CompileTask:
C2:398266147 1679  !   org.apache.jasper.compiler.Compiler.isOutDated(Z)Z (485 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x66870c00 JavaThread "http-8080-4" daemon [_thread_blocked, id=4856, stack(0x6b6a0000,0x6b710000)]
  0x66870800 JavaThread "http-8080-3" daemon [_thread_blocked, id=5684, stack(0x6b170000,0x6b1e0000)]
  0x66870000 JavaThread "ajp-8018-5" daemon [_thread_blocked, id=3532, stack(0x6b100000,0x6b170000)]
  0x6686fc00 JavaThread "ajp-8018-4" daemon [_thread_blocked, id=1400, stack(0x6b050000,0x6b0c0000)]
  0x6686f400 JavaThread "ajp-8018-3" daemon [_thread_blocked, id=2804, stack(0x6afe0000,0x6b050000)]
  0x6686ec00 JavaThread "http-8080-2" daemon [_thread_blocked, id=3940, stack(0x68fd0000,0x69040000)]
  0x6686e800 JavaThread "http-8080-1" daemon [_thread_blocked, id=5168, stack(0x68f60000,0x68fd0000)]
  0x6686e000 JavaThread "ajp-8018-2" daemon [_thread_blocked, id=4260, stack(0x68ef0000,0x68f60000)]
  0x6686dc00 JavaThread "ajp-8018-1" daemon [_thread_blocked, id=5476, stack(0x68e80000,0x68ef0000)]
  0x6686d400 JavaThread "ajp-8018-Acceptor-0" daemon [_thread_in_native, id=2640, stack(0x68e10000,0x68e80000)]
  0x6686d000 JavaThread "ajp-8018-CometPoller-7" daemon [_thread_blocked, id=4672, stack(0x68da0000,0x68e10000)]
  0x6686c800 JavaThread "ajp-8018-CometPoller-6" daemon [_thread_blocked, id=4892, stack(0x68d30000,0x68da0000)]
  0x6686c000 JavaThread "ajp-8018-CometPoller-5" daemon [_thread_blocked, id=1836, stack(0x68cc0000,0x68d30000)]
  0x6686bc00 JavaThread "ajp-8018-CometPoller-4" daemon [_thread_blocked, id=5452, stack(0x68c50000,0x68cc0000)]
  0x6686b400 JavaThread "ajp-8018-CometPoller-3" daemon [_thread_blocked, id=584, stack(0x68be0000,0x68c50000)]
  0x66fc7000 JavaThread "ajp-8018-CometPoller-2" daemon [_thread_blocked, id=1932, stack(0x68b70000,0x68be0000)]
  0x66fc6800 JavaThread "ajp-8018-CometPoller-1" daemon [_thread_blocked, id=5540, stack(0x68b00000,0x68b70000)]
  0x66fc6000 JavaThread "ajp-8018-CometPoller-0" daemon [_thread_blocked, id=2728, stack(0x68a90000,0x68b00000)]
  0x66fc5c00 JavaThread "ajp-8018-Poller-7" daemon [_thread_blocked, id=5104, stack(0x68a20000,0x68a90000)]
  0x66fc5400 JavaThread "ajp-8018-Poller-6" daemon [_thread_blocked, id=3756, stack(0x689b0000,0x68a20000)]
  0x66fc5000 JavaThread "ajp-8018-Poller-5" daemon [_thread_blocked, id=4888, stack(0x68940000,0x689b0000)]
  0x66fc4800 JavaThread "ajp-8018-Poller-4" daemon [_thread_in_native, id=3924, stack(0x688d0000,0x68940000)]
  0x66fc4400 JavaThread "ajp-8018-Poller-3" daemon [_thread_blocked, id=2632, stack(0x68860000,0x688d0000)]
  0x66fc3c00 JavaThread "ajp-8018-Poller-2" daemon [_thread_blocked, id=3792, stack(0x687f0000,0x68860000)]
  0x66fc3400 JavaThread "ajp-8018-Poller-1" daemon [_thread_blocked, id=4840, stack(0x68380000,0x683f0000)]
  0x66fc3000 JavaThread "ajp-8018-Poller-0" daemon [_thread_blocked, id=5424, stack(0x68310000,0x68380000)]
  0x66fc2800 JavaThread "http-8080-Acceptor-0" daemon [_thread_in_native, id=5532, stack(0x682a0000,0x68310000)]
  0x66fc2400 JavaThread "http-8080-Sendfile-0" daemon [_thread_blocked, id=2560, stack(0x68230000,0x682a0000)]
  0x66fc1c00 JavaThread "http-8080-CometPoller-7" daemon [_thread_blocked, id=4264, stack(0x681c0000,0x68230000)]
  0x66fc1800 JavaThread "http-8080-CometPoller-6" daemon [_thread_blocked, id=5948, stack(0x68150000,0x681c0000)]
  0x66fc1000 JavaThread "http-8080-CometPoller-5" daemon [_thread_blocked, id=3768, stack(0x680e0000,0x68150000)]
  0x66fc0800 JavaThread "http-8080-CometPoller-4" daemon [_thread_blocked, id=5772, stack(0x68070000,0x680e0000)]
  0x66fc0400 JavaThread "http-8080-CometPoller-3" daemon [_thread_blocked, id=564, stack(0x68000000,0x68070000)]
  0x66fbfc00 JavaThread "http-8080-CometPoller-2" daemon [_thread_blocked, id=1192, stack(0x67f90000,0x68000000)]
  0x66fbf800 JavaThread "http-8080-CometPoller-1" daemon [_thread_blocked, id=2776, stack(0x67f20000,0x67f90000)]
  0x66c2f800 JavaThread "http-8080-CometPoller-0" daemon [_thread_blocked, id=1012, stack(0x67eb0000,0x67f20000)]
  0x66dbe400 JavaThread "http-8080-Poller-7" daemon [_thread_blocked, id=3288, stack(0x67e40000,0x67eb0000)]
  0x66d9d800 JavaThread "http-8080-Poller-6" daemon [_thread_blocked, id=3684, stack(0x67dd0000,0x67e40000)]
  0x669af000 JavaThread "http-8080-Poller-5" daemon [_thread_blocked, id=4244, stack(0x67d60000,0x67dd0000)]
  0x6701b000 JavaThread "http-8080-Poller-4" daemon [_thread_blocked, id=2604, stack(0x67a90000,0x67b00000)]
  0x66f4f000 JavaThread "http-8080-Poller-3" daemon [_thread_blocked, id=2276, stack(0x67a20000,0x67a90000)]
  0x66509000 JavaThread "http-8080-Poller-2" daemon [_thread_blocked, id=5324, stack(0x679b0000,0x67a20000)]
  0x66c5a400 JavaThread "http-8080-Poller-1" daemon [_thread_blocked, id=2628, stack(0x678f0000,0x67960000)]
  0x66b9a800 JavaThread "http-8080-Poller-0" daemon [_thread_blocked, id=5576, stack(0x67850000,0x678c0000)]
  0x668b4000 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=3152, stack(0x677e0000,0x67850000)]
  0x66853000 JavaThread "GC Daemon" daemon [_thread_blocked, id=1636, stack(0x675e0000,0x67650000)]
  0x00f9c800 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=3184, stack(0x666f0000,0x66760000)]
=>0x00f99800 JavaThread "C2 CompilerThread1" daemon [_thread_in_native, id=1820, stack(0x66680000,0x666f0000)]
  0x00f98400 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=3932, stack(0x66610000,0x66680000)]
  0x00f96c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=2612, stack(0x663a0000,0x66410000)]
  0x00f95c00 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=5128, stack(0x66330000,0x663a0000)]
  0x00f68800 JavaThread "Finalizer" daemon [_thread_blocked, id=4868, stack(0x662c0000,0x66330000)]
  0x00f62000 JavaThread "Reference Handler" daemon [_thread_blocked, id=5500, stack(0x66250000,0x662c0000)]
  0x00e5a000 JavaThread "main" [_thread_in_native, id=2800, stack(0x00d70000,0x00de0000)]

Other Threads:
  0x00f5e800 VMThread [stack: 0x661e0000,0x66250000] [id=4608]
  0x00f9d000 WatcherThread [stack: 0x66760000,0x667d0000] [id=3600]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap
 PSYoungGen      total 368192K, used 215481K [0x46860000, 0x65730000, 0x65c60000)
  eden space 230016K, 84% used [0x46860000,0x5263eae8,0x54900000)
  from space 138176K, 15% used [0x5cfb0000,0x5e43f968,0x656a0000)
  to   space 137920K, 0% used [0x54900000,0x54900000,0x5cfb0000)
 PSOldGen        total 349568K, used 320646K [0x08060000, 0x1d5c0000, 0x46860000)
  object space 349568K, 91% used [0x08060000,0x1b981b60,0x1d5c0000)
 PSPermGen       total 33664K, used 33565K [0x04060000, 0x06140000, 0x08060000)
  object space 33664K, 99% used [0x04060000,0x061277e8,0x06140000)

Code Cache  [0x00fa0000, 0x01570000, 0x03fa0000)
 total_blobs=1881 nmethods=1658 adapters=176 free_code_cache=44292736 largest_free_block=10112

Dynamic libraries:
0x00400000 - 0x00417000     D:\Kappa\apache-tomcat-6.0.36\bin\tomcat6.exe
0x77460000 - 0x77587000     C:\Windows\system32\ntdll.dll
0x76490000 - 0x7656c000     C:\Windows\system32\kernel32.dll
0x75ef0000 - 0x75fb6000     C:\Windows\system32\ADVAPI32.dll
0x75fc0000 - 0x76083000     C:\Windows\system32\RPCRT4.dll
0x76570000 - 0x77080000     C:\Windows\system32\SHELL32.dll
0x762d0000 - 0x7637a000     C:\Windows\system32\msvcrt.dll
0x75d30000 - 0x75d7b000     C:\Windows\system32\GDI32.dll
0x75e50000 - 0x75eed000     C:\Windows\system32\USER32.dll
0x76270000 - 0x762c9000     C:\Windows\system32\SHLWAPI.dll
0x775d0000 - 0x775ee000     C:\Windows\system32\IMM32.DLL
0x775f0000 - 0x776b8000     C:\Windows\system32\MSCTF.dll
0x77590000 - 0x77599000     C:\Windows\system32\LPK.DLL
0x75d80000 - 0x75dfd000     C:\Windows\system32\USP10.dll
0x748d0000 - 0x74a6e000     C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_6.0.6002.18005_none_5cb72f96088b0de0\comctl32.dll
0x6db90000 - 0x6dfea000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\server\jvm.dll
0x743e0000 - 0x74412000     C:\Windows\system32\WINMM.dll
0x77240000 - 0x77385000     C:\Windows\system32\ole32.dll
0x771b0000 - 0x7723d000     C:\Windows\system32\OLEAUT32.dll
0x74380000 - 0x743b9000     C:\Windows\system32\OLEACC.dll
0x7c340000 - 0x7c396000     C:\Windows\system32\MSVCR71.dll
0x75ae0000 - 0x75b0c000     C:\Windows\system32\apphelp.dll
0x6d870000 - 0x6d87c000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\verify.dll
0x6d3d0000 - 0x6d3ef000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\java.dll
0x6d8b0000 - 0x6d8bf000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\zip.dll
0x6d600000 - 0x6d609000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\management.dll
0x673d0000 - 0x674d7000     D:\Kappa\apache-tomcat-6.0.36\bin\tcnative-1.dll
0x75cd0000 - 0x75cd7000     C:\Windows\system32\PSAPI.DLL
0x775a0000 - 0x775cd000     C:\Windows\system32\WS2_32.dll
0x760a0000 - 0x760a6000     C:\Windows\system32\NSI.dll
0x75300000 - 0x7533b000     C:\Windows\system32\MSWSOCK.dll
0x75060000 - 0x7509b000     C:\Windows\system32\rsaenh.dll
0x6d850000 - 0x6d858000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\sunmscapi.dll
0x755e0000 - 0x756d2000     C:\Windows\system32\CRYPT32.dll
0x75740000 - 0x75752000     C:\Windows\system32\MSASN1.dll
0x75b90000 - 0x75bae000     C:\Windows\system32\USERENV.dll
0x75b70000 - 0x75b84000     C:\Windows\system32\Secur32.dll
0x74f00000 - 0x74f05000     C:\Windows\System32\wshtcpip.dll
0x6d6b0000 - 0x6d6c3000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\net.dll
0x75360000 - 0x75365000     C:\Windows\System32\wship6.dll
0x74c20000 - 0x74c2f000     C:\Windows\system32\NLAapi.dll
0x75540000 - 0x75559000     C:\Windows\system32\IPHLPAPI.DLL
0x75500000 - 0x75535000     C:\Windows\system32\dhcpcsvc.DLL
0x75780000 - 0x757ac000     C:\Windows\system32\DNSAPI.dll
0x754f0000 - 0x754f7000     C:\Windows\system32\WINNSI.DLL
0x754c0000 - 0x754e2000     C:\Windows\system32\dhcpcsvc6.DLL
0x74290000 - 0x7429f000     C:\Windows\system32\napinsp.dll
0x74270000 - 0x74278000     C:\Windows\System32\winrnr.dll
0x75e00000 - 0x75e49000     C:\Windows\system32\WLDAP32.dll
0x74280000 - 0x74286000     C:\Windows\system32\rasadhlp.dll
0x003d0000 - 0x003dd000     D:\Kappa\Java\jdk1.6.0_43\bin\ntlmauth.dll
0x75410000 - 0x75417000     C:\Windows\system32\credssp.dll
0x750d0000 - 0x75114000     C:\Windows\system32\schannel.dll
0x759a0000 - 0x75a16000     C:\Windows\system32\NETAPI32.dll
0x752c0000 - 0x752f7000     C:\Windows\system32\msv1_0.dll
0x757b0000 - 0x757c1000     C:\Windows\system32\cryptdll.dll
0x6d0b0000 - 0x6d1fc000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\awt.dll
0x707c0000 - 0x70802000     C:\Windows\system32\WINSPOOL.DRV
0x740d0000 - 0x74155000     C:\Windows\WinSxS\x86_microsoft.windows.common-controls_6595b64144ccf1df_5.82.6001.18000_none_886786f450a74a05\COMCTL32.dll
0x00920000 - 0x0092b000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_util.dll
0x6b250000 - 0x6b69e000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\mlib_jai_mmx.dll
0x6d220000 - 0x6d24f000     D:\Kappa\Java\jdk1.6.0_43\jre\bin\cmm.dll

VM Arguments:
jvm_args: -Dcatalina.base=D:\Kappa\apache-tomcat-6.0.36 -Dcatalina.home=D:\Kappa\apache-tomcat-6.0.36 -Djava.endorsed.dirs=D:\Kappa\apache-tomcat-6.0.36\endorsed -Djava.io.tmpdir=D:\Kappa\apache-tomcat-6.0.36\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=D:\Kappa\apache-tomcat-6.0.36\conf\logging.properties -Xms512m -Xmx1500m 
java_command: <unknown>
Launcher Type: generic

Environment Variables:
JAVA_HOME=D:\Kappa\Java\jdk1.6.0_43
CLASSPATH=D:\Kappa\apache-tomcat-6.0.36\lib\servlet-api.jar
PATH=C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;D:\Kappa\Java\jdk1.6.0_43\bin;C:\Program Files\Windows Imaging\
USERNAME=SVC-UPSLKappa-DIA
OS=Windows_NT
PROCESSOR_IDENTIFIER=x86 Family 6 Model 23 Stepping 6, GenuineIntel



---------------  S Y S T E M  ---------------

OS: Windows Server 2008 Build 6002 Service Pack 2

CPU:total 2 (1 cores per cpu, 1 threads per core) family 6 model 23 stepping 6, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1

Memory: 4k page, physical 2095900k(292724k free), swap 4467300k(2557472k free)

vm_info: Java HotSpot(TM) Server VM (20.14-b01) for windows-x86 JRE (1.6.0_43-b01), built on Mar  1 2013 02:47:24 by "java_re" with MS VC++ 7.1 (VS2003)

time: Tue Oct 01 08:51:47 2013
elapsed time: 398205 seconds

回答by Martin Kersten

Use a profiler like VisualVM. Connect it to monitor the memory use of your tomcat server (remote configuration). It will constantly monitor and show you your heap allocation. You can also see the objects being hold in memory.

使用像VisualVM这样的分析器。连接它来监控你的tomcat服务器的内存使用（远程配置）。它会持续监控并向您显示您的堆分配。您还可以看到保存在内存中的对象。

If this happens you have either hit the memory limit of the process (you will see the total used amount in visual vm). It might be even a memory leak or a bug in the JVM. Check if you are using the 64 bit version. But first check out the object creation and heap using VisualVM (or similar). You can tell by the graph if your consumption goes up in a different way.

如果发生这种情况，您要么达到了进程的内存限制（您将在视觉虚拟机中看到总使用量）。它甚至可能是内存泄漏或 JVM 中的错误。检查您是否使用的是 64 位版本。但首先使用 VisualVM（或类似方法）检查​​对象创建和堆。您可以通过图表判断您的消费量是否以不同方式上升。

You can even compare the heap usage pattern you gain using the previous used Java version. This way you can compare the memory consumption of the old working setup with the new problematic one.

您甚至可以比较使用以前使用的 Java 版本获得的堆使用模式。通过这种方式，您可以将旧工作设置的内存消耗与新的有问题的设置进行比较。

回答by Mark Thomas

The process heap != the java heap.

进程堆 != java 堆。

On 32-bit Windows you have (unless you set some boot switches) a 2GB maximum per process. This 2GB has to support:

在 32 位 Windows 上（除非您设置了一些启动开关）每个进程最大 2GB。这 2GB 必须支持：

• Java Heap (set with Xms, Xmx)
• PermGen
• Thread stacks
• GC
• Code cache
• Native JVM code

• Java 堆（用 Xms、Xmx 设置）
• 永久代
• 线程栈
• GC
• 代码缓存
• 原生 JVM 代码

and probably a couple of things I have forgotten.

可能还有几件事我已经忘记了。

If the total of all of the above exceeds 2GB the process crashes. This is what has happended in your case.

如果以上所有内容的总和超过 2GB，则进程崩溃。这就是你的情况。

The error message tells you quite clearly what you need to do. Pick one of the options and implement it.

错误消息非常清楚地告诉您需要做什么。选择其中一个选项并实施它。