java8“java.lang.OutOfMemoryError：元空间”

java8 "java.lang.OutOfMemoryError: Metaspace"

提问by Jane

After switching our java application (services running on Tomcat) JRE from Java 7 to Java 8, we started to see java.lang.OutOfMemoryError: Metaspaceafter running a few days with high traffic volume.

在将我们的 Java 应用程序（在 Tomcat 上运行的服务）JRE 从 Java 7 切换到 Java 8 后，我们开始java.lang.OutOfMemoryError: Metaspace运行几天后看到高流量。

Heap usage was OK. Metaspace jumps after sometime when the same code flow was executed during performance testing.

堆使用正常。在性能测试期间执行相同的代码流一段时间后，元空间会跳转。

What could be possible causes of the metaspace memory issue?

元空间内存问题的可能原因是什么？

Current settings is:

当前设置是：

-server -Xms8g -Xmx8g -XX:MaxMetaspaceSize=3200m  -XX:+UseParNewGC 
-XX:+UseConcMarkSweepGC -XX:MaxGCPauseMillis=1000 
-XX:+DisableExplicitGC -XX:+PrintGCDetails 
-XX:-UseAdaptiveSizePolicy -XX:SurvivorRatio=7 -XX:NewSize=5004m 
-XX:MaxNewSize=5004m -XX:MaxTenuringThreshold=12 
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+PrintFlagsFinal  
-XX:+PrintGCDateStamps -XX:+PrintTenuringDistribution 
-XX:+PrintGCCause -XX:+PrintAdaptiveSizePolicy 
-XX:+UseGCLogFileRotation -XX:NumberOfGCLogFiles=3 -XX:GCLogFileSize=200M 

Also the application has heavy use of reflection. Also we use a custom class loader. All of them were working fine in java 7.

此外，该应用程序大量使用反射。我们还使用自定义类加载器。所有这些都在 java 7 中正常工作。

回答by dillip pattnaik

I assume you can create the issue with same request (Set of requests) over a period of time. It is a good thing that you have MaxMetaspaceSize defined, otherwise app will use native memory till it runs out to grow. But i will start with following steps:

我假设您可以在一段时间内使用相同的请求（请求集）创建问题。定义 MaxMetaspaceSize 是一件好事，否则应用程序将使用本机内存，直到它用完为止。但我将从以下步骤开始：

1. Check if your number of classes that are loaded in JVM keeps growing for same request when you send it to the server multiple times. If yes, you may be creating dynamic classes which would cause growth in classes loaded in metaspace. Well how to check the number of classes loaded, you can use visualvm to connect to the server using JMX or run locally to simulate. I will mention steps for local , but for remote attaching JMX , you should add following to the JVM parameters to application and start it and remote connect on port 9999 and with -XX:+UnlockDiagnosticVMOptions.

1. 当您多次将同一请求发送到服务器时，请检查在 JVM 中加载的类的数量是否不断增加。如果是，您可能正在创建动态类，这会导致元空间中加载的类增加。那么如何查看加载的类的数量，可以使用visualvm使用JMX连接服务器或者本地运行模拟。我将提到本地的步骤，但对于远程附加 JMX，您应该将以下内容添加到应用程序的 JVM 参数中并启动它并在端口 9999 和 -XX:+UnlockDiagnosticVMOptions 上远程连接。

   -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UnlockDiagnosticVMOptions

   -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -XX:+UnlockDiagnosticVMOptions

Once you have the visualvm (jvisualvm) connected to the JVM, click on monitor and then see the number of classes loaded. There you can monitor the heap as well as metaspace. But i will add the other tools to closely monitor the metaspace.

一旦将visualvm（jvisualvm）连接到JVM，点击monitor，然后查看加载的类的数量。在那里您可以监视堆以及元空间。但我会添加其他工具来密切监视元空间。

1. Also once you're connected to the jvm, you may want to take a heap snapshot and find out the classes loaded using OQL. So to before you take a heap dump, stop the requests to the server , so you dont catch any inflight request/executing code and their associated objects, but it is not necessary. So after running same set of requests multiple times , inside visualvm , in the "monitor" space, click on "Heap Dump" on the right top". Then open/load the snapshot, and you would see the option to OQL console. And you would see some predefined OQL queries on the right bottom panel under permgen analysis. Run the query named "classloader loaded class histogram", i guess that will give the count of classes loaded by each classloader. You can use it to find out which classloader is loading classes.

2. 此外，一旦您连接到 jvm，您可能需要拍摄堆快照并找出使用 OQL 加载的类。因此，在进行堆转储之前，请停止对服务器的请求，这样您就不会捕获任何进行中的请求/执行代码及其相关对象，但这不是必需的。因此，在多次运行同一组请求后，在 visualvm 中，在“监视器”空间中，单击右上角的“堆转储”。然后打开/加载快照，您将看到 OQL 控制台的选项。和您会在 permgen 分析下的右下面板上看到一些预定义的 OQL 查询。运行名为“类加载器加载的类直方图”的查询，我想这将给出每个类加载器加载的类的数量。您可以使用它来找出哪个类加载器正在加载类。

select map(sort(map(heap.objects('java.lang.ClassLoader'), '{loader: it, count: it.classes.elementCount }'), 'lhs.count < rhs.count'), 'toHtml(it) + "
"')

select map(sort(map(heap.objects('java.lang.ClassLoader'), '{loader: it, count: it.classes.elementCount }'), 'lhs.count < rhs.count'), 'toHtml (它) + "
"')

But the query above that named "classloader loaded class" will be slow which will actually show the classes loaded by each classloader.

但是上面名为“类加载器加载的类”的查询会很慢，它实际上会显示每个类加载器加载的类。

select { loader: cl,
             classes: filter(map(cl.classes.elementData, 'it'), 'it != null') }
    from instanceof java.lang.ClassLoader cl

1. Then try to trace down the growth in metaspace area. Now we will use jconsole and something new that java has: jmc (java mission control). You may use jconsole to connect to the jvm (local or remote) and once you are connected go the memory tab and you can monitor the non heap growth there, which should have metaspace and code cache and compressed class space. And now connect

3. 然后尝试追踪元空间区域的增长。现在我们将使用 jconsole 和 java 有的新东西：jmc（java 任务控制）。您可以使用 jconsole 连接到 jvm（本地或远程），一旦连接到内存选项卡，您就可以监控那里的非堆增长，它应该有元空间和代码缓存以及压缩的类空间。现在连接

jmc

JMC

to connect to the VM and then once you are connected click on "Diagnostic commands" in the JMC which is on the right top side. Since we have enabled UnlockDiagnosticVMOptions , GC.class_stats could be executed. You may want to run it with show all columns and print in csv. So the command will look like:

连接到虚拟机，然后在连接后单击右上角的 JMC 中的“诊断命令”。由于我们启用了 UnlockDiagnosticVMOptions ，因此可以执行 GC.class_stats 。您可能希望通过显示所有列运行它并在 csv 中打印。所以命令看起来像：

GC.class_stats -all=true -csv=true

And then you can compare the class stats over different periods and find out which classes are causing trouble(metaspace growth) or which classes have related information( method/method data) in metaspace. How to analyze the csv outputs collected over at time: well i would take that csv and load it up in two similar table(representing csv) in a database or some other place to compare GC.class_stats csv outputs where i can run some SQL or any other analyzing tools. That would give a better idea of what is exactly growing in metaspace. The GC class stats has following columns:

然后你可以比较不同时期的类统计数据，找出哪些类导致问题（元空间增长）或哪些类在元空间中有相关信息（方法/方法数据）。如何分析当时收集的 csv 输出：好吧，我会将该 csv 加载到数据库或其他地方的两个类似表（代表 csv）中，以比较 GC.class_stats csv 输出，我可以在其中运行一些 SQL 或任何其他分析工具。这样可以更好地了解元空间中究竟增长了什么。GC 类统计信息包含以下列：

Index,Super,InstSize,InstCount,InstBytes,Mirror,KlassBytes,K_secondary_supers,VTab,ITab,OopMap,IK_methods,IK_method_ordering,IK_default_methods,IK_default_vtable_indices,IK_local_interfaces,IK_transitive_interfaces,IK_fields,IK_inner_classes,IK_signers,class_annotations,class_type_annotations,fields_annotations,fields_type_annotations,methods_annotations,methods_parameter_annotations,methods_type_annotations,methods_default_annotations,annotations,Cp,CpTags,CpCache,CpOperands,CpRefMap,CpAll,MethodCount,MethodBytes,ConstMethod,MethodData,StackMap,Bytecodes,MethodAll,ROAll,RWAll,Total,ClassName,ClassLoader

指数，超级，InstSize，InstCount，InstBytes，镜子，KlassBytes，K_secondary_supers，VTab，ITAB，OopMap，IK_methods，IK_method_ordering，IK_default_methods，IK_default_vtable_indices，IK_local_interfaces，IK_transitive_interfaces，IK_fields，IK_inner_classes，IK_signers，class_annotations，class_type_annotations，fields_annotations，fields_type_annotations，methods_annotations， methods_parameter_annotations,methods_type_annotations,methods_default_annotations,annotations,Cp,CpTags,CpCache,CpOperands,CpRefMap,CpAll,MethodCount,MethodBytes,ConstMethod,MethodData,StackMap,Bytecodes,MethodAll,ROAll,RWAll,Loader,ClassName,Class

Hope it helps. Also it appears the bug may be in Java 8 if it does not cause any leak in 1.7.

希望能帮助到你。此外，如果它不会在 1.7 中导致任何泄漏，则该错误可能出现在 Java 8 中。

Also the classes will not get unloaded from metaspace if any one is holding any reference to classloader. If you know your classloaders are supposed to be GCed and no one should hold the reference to your class loader, you can go back to heap dump in visualvm and click on class loader instance and right click to find "nearest GC root" which will tell you who is holding the reference to the classloaders.

如果任何人持有对类加载器的任何引用，这些类也不会从元空间中卸载。如果你知道你的类加载器应该被 GC 并且没有人应该持有对你的类加载器的引用，你可以返回到 Visualvm 中的堆转储并单击类加载器实例并右键单击以找到“最近的 GC 根”，它将告诉您持有对类加载器的引用。

回答by user1550159

we had similar issue and the root cause was 60K class files are getting loaded into metaspace memory, but nothing getting unloaded.Adding below JVM arg fixed out issue.

我们遇到了类似的问题，根本原因是 60K 类文件被加载到元空间内存中，但没有被卸载。在 JVM arg 下面添加修复了问题。

-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

https://issues.apache.org/jira/browse/CXF-2939

https://issues.apache.org/jira/browse/CXF-2939

Hope this helps.

希望这可以帮助。

回答by Dove

also if some auto deploy like tomcat, do NOT keep backups in tomcat\webapps otherwise it possibly try to load the backup and collide with those resources.

此外，如果某些自动部署如 tomcat，请勿将备份保存在 tomcat\webapps 中，否则它可能会尝试加载备份并与这些资源发生冲突。