Java ConcurrentHashMap充分利用内存（Tomcat下）

Memory Fully utilized by Java ConcurrentHashMap (under Tomcat)

提问by Reusable

This is a memory stack (serves as a cache) that consist of nothing but a static ConcurrentHashMap (CHM).

这是一个内存堆栈（用作缓存），仅由静态 ConcurrentHashMap (CHM) 组成。

All incoming HTTP request data are store in this ConcurrentHashMap. And there is a asynch scheduler process that takes the data from the same ConcurrentHashMap and remove the key.value after storing them into the Database.

所有传入的 HTTP 请求数据都存储在此 ConcurrentHashMap 中。并且有一个异步调度程序进程，它从同一个 ConcurrentHashMap 中获取数据，并在将它们存储到数据库中后删除 key.value。

This system runs fine and smooth but just discover under following criteria, the memory was fully utilized (2.5GB) and all CPU time was taken to perform GC:

该系统运行良好且流畅，但在以下条件下发现，内存已被充分利用（2.5GB）并且所有 CPU 时间都用于执行 GC：

-concurrent http hit of 1000/s

-并发 http 命中 1000/s

-maintain the same concurrent hit for a period of 15 minutes

- 保持相同的并发命中 15 分钟

The asynch process log the remaining size of the CHM everytime it writes to database. The CHM.size() maintain at around Min:300 to Max:3500

异步进程每次写入数据库时​​都会记录 CHM 的剩余大小。CHM.size() 保持在 Min:300 到 Max:3500 左右

I thought there is a Memory Leak on this application. so i used Eclipse MAT to look at the Heap Dump. After running the Suspect Report, i got these comments from MAT:

我认为此应用程序存在内存泄漏。所以我使用 Eclipse MAT 来查看堆转储。运行可疑报告后，我从 MAT 那里得到了这些评论：

One instance of "org.apache.catalina.session.StandardManager" loaded by "org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes. The memory is accumulated in one instance of "java.util.concurrent.ConcurrentHashMap$Segment[]" loaded by "".

一个由“org.apache.catalina.loader.StandardClassLoader @ 0x853f0280”加载的“org.apache.catalina.session.StandardManager”实例占用2,135,429,456（94.76%）个字节。内存累积在一个由“”加载的“java.util.concurrent.ConcurrentHashMap$Segment[]”实例中。

3,646,166 instances of java.util.concurrent.ConcurrentHashMap$Segment retain >= 2,135,429,456 bytes.

and

和

Length    # Objects      Shallow Heap      Retained Heap 
0         3,646,166      482,015,968       >= 2,135,429,456 

The length 0 above i translate it as empty length record inside the CHM (each time i call CHM.remove() method). It is consistent to the number of record inside the database, 3,646,166 records was inside the database when this dump was created

上面的长度 0 我将其转换为 CHM 内的空长度记录（每次我调用 CHM.remove() 方法时）。与数据库内的记录数一致，创建此转储时数据库内有3,646,166条记录

The strange scenario is: if i pause the stress test, the utilization in Heap Memory will gradually release down to 25MB.This takes about 30-45 minutes. i have re-simulate this application and the curves looks similar to the VisualVM Graph below:

奇怪的情况是：如果我暂停压力测试，堆内存的利用率会逐渐释放到 25MB。这大约需要 30-45 分钟。我重新模拟了这个应用程序，曲线看起来类似于下面的 VisualVM Graph：

Heres the questions:

以下是问题：

1) Does this looks like a Memory Leak?

1) 这看起来像内存泄漏吗？

2) Each remove call remove(Object key, Object value)to remove a <key:value>from CHM, does that removed object get GC?

2) 每个从 CHM 中remove(Object key, Object value)删除 a 的删除调用<key:value>，该删除的对象是否获得 GC？

3) Is this something to do with the GC settings? i have added the following GC parameters but without help:

3）这与GC设置有关吗？我添加了以下 GC 参数，但没有帮助：

-XX:+UseParallelGC

-XX:+UseParallelOldGC

-XX:GCTimeRatio=19

-XX:+PrintGCTimeStamps

-XX:ParallelGCThreads=6

-verbose:gc

4) Any idea to resolve this is very much appreciated! :)

4）任何解决这个问题的想法都非常感谢！:)

NEW5) Could it be possible because all my reference are hard reference? My understanding is as long as the HTTP session is ended, all those variables that is not static are now available for GC.

新5) 有没有可能因为我所有的参考都是硬参考？我的理解是，只要 HTTP 会话结束，所有那些非静态变量现在都可用于 GC。

NEWNote I tried replace the CHM with ehcache 2.2.0, but i get the same OutOfMemoryException problem. i suppose ehcache is also using ConcurrentHashMap.

新注意我尝试用 ehcache 2.2.0 替换 CHM，但我遇到了相同的 OutOfMemoryException 问题。我想 ehcache 也在使用 ConcurrentHashMap。

Server Spec:

服务器规格：

-Xeon Quad core, 8 threads.

-至强四核，8 线程。

-4GB Memory

-4GB 内存

-Windows 2008 R2

-Windows 2008 R2

-Tomcat 6.0.29

-Tomcat 6.0.29

回答by Jesper

1) Does this looks like a Memory Leak?

1) 这看起来像内存泄漏吗？

Yes, if the application keeps on putting objects in the map and never removes them, then that could very well be a memory leak.

是的，如果应用程序不断将对象放入映射中并且从不删除它们，那么这很可能是内存泄漏。

2) Each remove call remove(Object key, Object value) to remove a from CHM, does that removed object get GC?

2) 每次删除调用 remove(Object key, Object value) 从 CHM 中删除一个，删除的对象是否获得 GC？

Objects can only be garbage collected if there is no live (running) thread that has a reference to them. The map is only one place where there's a reference to the object. There could still be other places that have references to the same object. But keeping the object in the map will prevent it from being garbage collected.

只有在没有引用对象的活动（正在运行）线程时，才能对对象进行垃圾回收。地图只是一个引用对象的地方。可能还有其他地方引用了同一个对象。但是将对象保留在地图中将防止它被垃圾收集。

3) Is this something to do with the GC settings?

3）这与GC设置有关吗？

No; if an object is referenced, it cannot be garbage collected; it doesn't matter how you tweak the garbage collector.

不; 如果一个对象被引用，它就不能被垃圾回收；你如何调整垃圾收集器并不重要。

回答by Reusable

This problem has bug me for a bad 7 days! And finally i found out the real problem! Below are the tasks on what i have tried but failed to solve the OutOfMemory Exception:

这个问题困扰了我 7 天！最后我发现了真正的问题！以下是我尝试过但未能解决 OutOfMemory 异常的任务：

-change from using concurrenthashmap to ehcache. (turns out ehcache is also using ConcurrentHashMap)

- 从使用 concurrenthashmap 到 ehcache。（结果 ehcache 也在使用 ConcurrentHashMap）

-change all the hard reference to Soft Reference

- 将所有硬引用更改为软引用

-Override the AbstractMap along side with concurrnetHashMap as per suggest by Dr. Heinz M. Kabutz

-按照Heinz M. Kabutz 博士的建议，将 AbstractMap 与 concurrnetHashMap 一起覆盖

The million dollar question is really "why 30-45 minutes later, memory starting to release back to the heap pool?"

百万美元的问题真的是“为什么 30-45 分钟后，内存开始释放回堆池？”

The actual root cause was because there is something else still holding the actual variable session, and the culprit is the http session within tomcat is still active! Hence, even though the http session was completed, but if the timeout setting is 30 minutes, tomcat will hold the session information for 30 minutes before JVM can GC those. Problem solve immediately after changing the timeout setting to 1 minute as testing.

真正的根本原因是因为还有其他东西仍然持有实际的变量会话，而罪魁祸首是 tomcat 中的 http 会话仍然处于活动状态！因此，即使 http 会话已完成，但如果超时设置为 30 分钟，tomcat 将在 JVM GC 之前将会话信息保留 30 分钟。将超时设置更改为 1 分钟作为测试后立即解决问题。

$tomcat_folder\conf\web.xml

<session-config>
    <session-timeout>1</session-timeout>
</session-config>

Hope this will help anyone out there with similar problem.

希望这会帮助任何有类似问题的人。

回答by Hendy Irawan

I think you're using too muchsession datathat won't fit at oncein memory. Try this one:

我认为您使用了过多的会话数据，无法一次性放入内存。试试这个：

1. Edit bin/setenv.shor wherever the JVM argsare set on your Tomcat launcher :

   Append -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true

   e.g.

   # Default Java options
   if [ -z "$JAVA_OPTS" ]; then
           JAVA_OPTS="-server -Djava.awt.headless=true -XX:MaxPermSize=384m -Xmx1024m -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true"
   fi
   

2. Edit conf/context.xml, before </Context>add this:

   <Manager className="org.apache.catalina.session.PersistentManager"
           maxIdleBackup="60" maxIdleSwap="300">
       <Store className="org.apache.catalina.session.FileStore"/>
   </Manager>
   

1. 编辑bin/setenv.sh或在Tomcat 启动器上设置JVM 参数的任何位置：

   附加 -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true

   例如

   # Default Java options
   if [ -z "$JAVA_OPTS" ]; then
           JAVA_OPTS="-server -Djava.awt.headless=true -XX:MaxPermSize=384m -Xmx1024m -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true"
   fi
   

2. 编辑conf/context.xml，在</Context>添加之前：

   <Manager className="org.apache.catalina.session.PersistentManager"
           maxIdleBackup="60" maxIdleSwap="300">
       <Store className="org.apache.catalina.session.FileStore"/>
   </Manager>
   

Restart Tomcat and your problem should be gone, since it'll store your sessions using the filesysteminstead.

重新启动 Tomcat，您的问题应该消失了，因为它将使用文件系统来存储您的会话。

In my view setting session-timeout = 1is a workaroundthat masks the root of problem, and is unusable in most apps where you actually need a big enough session-timeout. Our (Bippo's) apps usually have a session-timeoutof 2880minutes i.e. 2 days.

我认为设置session-timeout = 1是一个解决办法掩盖问题的根源，并在你真正需要一个足够大的大多数应用程序无法使用session-timeout。我们（Bippo的），通常应用程序都有一个session-timeout的2880分钟，即2天。

Reference: Tomcat 7.0 Session Manager Configuration

参考：Tomcat 7.0 会话管理器配置

回答by Zlelik

Of course, it is too late to answer, but just for other people who will find this question by search. It might be useful.

当然，现在回答已经太晚了，但只是为了其他会通过搜索找到这个问题的人。它可能有用。

These 2 links are very useful
https://issues.apache.org/bugzilla/show_bug.cgi?id=50685
http://wiki.apache.org/tomcat/OutOfMemory

这两个链接非常有用
https://issues.apache.org/bugzilla/show_bug.cgi?id=50685
http://wiki.apache.org/tomcat/OutOfMemory

Briefly, in most cases it is a wrong test or test software. When some custom software open URL, if this software cannot manage http session tomcat creates new session for each request. For example it is possible to check it with simple code, which can be added to JSP.

简而言之，在大多数情况下，它是错误的测试或测试软件。当某些自定义软件打开 URL 时，如果该软件无法管理 http 会话，tomcat 会为每个请求创建新的会话。例如，可以用简单的代码进行检查，可以将其添加到 JSP 中。

System.out.println("session id: " + session.getId());
System.out.println("session obj: " + session);
System.out.println("session getCreationTime: " + (new Date(session.getCreationTime())).toString());
System.out.println("session.getValueNames().length: " + session.getValueNames().length);

If session ID will be the same for one user from load test point of view, it is fine, if each request generates new session ID, that means testing software does not manage the sessions very well and test result does not represent load from real users.

如果从负载测试的角度来看，一个用户的会话 ID 是相同的，那很好，如果每个请求都生成新的会话 ID，这意味着测试软件没有很好地管理会话，测试结果不代表真实用户的负载.

For some application session.getValueNames().length also important, because For example, when normal user works it remain the same, but when load testing software do the same, it is grows. It also means, that load testing software does not represent real workload very well. In my case session.getValueNames().length for normal user was about 100, but qwith load testing software after 10 minutes is was about 500 and finally system crashes with the same OutOfMemory error and MAT shows the same:

对于某些应用程序 session.getValueNames().length 也很重要，因为例如，当普通用户工作时它保持不变，但是当负载测试软件做同样的事情时，它是增长的。这也意味着，负载测试软件并不能很好地代表真实的工作负载。在我的例子中，普通用户的 session.getValueNames().length 约为 100，但 10 分钟后的 qwith 负载测试软件约为 500，最终系统崩溃并出现相同的 OutOfMemory 错误，MAT 显示相同：

org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" occupies 2,135,429,456 (94.76%) bytes.

org.apache.catalina.loader.StandardClassLoader @ 0x853f0280" 占用 2,135,429,456 (94.76%) 个字节。

回答by user1819111

If you get this exception and are using spring boot version 1.4.4 RELEASE or lower, set value of property "server.session-timeout" in minutes, rather than what they suggest (seconds), so that the sessions on the heap will be cleaned in time. Or you can use a bean of EmbeddedServletContainerCustomizer but the provided value will be set in minutes.

如果您遇到此异常并且正在使用 Spring Boot 版本 1.4.4 RELEASE 或更低版本，请以分钟为单位设置属性“server.session-timeout”的值，而不是它们建议的值（秒），以便堆上的会话及时清洗。或者您可以使用 EmbeddedServletContainerCustomizer 的 bean，但提供的值将在几分钟内设置。

example (session timeout in 10 minutes): server.session-timeout=10 (set in properties file) container.setSessionTimeout(10, TimeUnit.SECONDS); (set in EmbeddedServletContainerCustomizer)

示例（会话超时 10 分钟）：server.session-timeout=10（在属性文件中设置） container.setSessionTimeout(10, TimeUnit.SECONDS); （在 EmbeddedServletContainerCustomizer 中设置）