Java Tomcat 自行关闭

Tomcat shutting down by itself

提问by PrecisionPete

I'm running Tomcat 7.0.53 on CentOS 6.5 64 bit and OpenJDK 1.7 64.

我在 CentOS 6.5 64 位和 OpenJDK 1.7 64 上运行 Tomcat 7.0.53。

I have several servers where occasionally - like randomly once per week - Tomcat will just gracefully shut itself down. I've been looking at this for months and cannot find the reason. The only pattern seems to be a long period of nothing going on then the shutdown.

我有几台服务器，偶尔 - 就像每周随机一次 - Tomcat 会优雅地关闭自己。我已经研究了几个月，但找不到原因。唯一的模式似乎是很长一段时间没有任何事情发生，然后关闭。

• I have upgraded to latest versions of the underlying tools (Tomcat, Java, etc)
• I am running with the default memory settings
• I have disabled the SHUTDOWN in the server.xml
• I have moved MySQL to another box to ensure Tomcat is the only substantial application (server also runs nginx)
• I have verified that I have no System.exit() in any of my code. I have not scanned the libs as I can't figure out how to do that. And I don't expect I will find anything there either. Why would a library exit anyway?
• I have enabled logging of garbage collection. But they seem pretty quick - like full GC in 0.15s

• 我已升级到最新版本的底层工具（Tomcat、Java 等）
• 我正在使用默认内存设置运行
• 我在 server.xml 中禁用了 SHUTDOWN
• 我已将 MySQL 移至另一个框以确保 Tomcat 是唯一的实质性应用程序（服务器也运行 nginx）
• 我已经确认我的任何代码中都没有 System.exit() 。我没有扫描库，因为我不知道如何做到这一点。而且我也不指望我会在那里找到任何东西。为什么图书馆会退出？
• 我已启用垃圾收集的日志记录。但它们看起来很快——就像 0.15 秒内的完整 GC

Here is the memory usage of one of the servers. Lots of RAM free.

这是其中一台服务器的内存使用情况。大量的免费内存。

                 total       used       free     shared    buffers     cached
    Mem:          2006        771       1234          0        176        281
    -/+ buffers/cache:        313       1692
    Swap:         2047          0       2047

Below is one of the events in the Catalina.out. You can see it starts then does nothing exciting for several hours. Then gracefully shuts itself down as if it was told to.

以下是 Catalina.out 中的事件之一。你可以看到它开始然后几个小时没有任何令人兴奋的事情。然后像被告知一样优雅地关闭自己。

I've researched this to death and have not bee able to get a handle on this.

我已经对此进行了研究，但一直无法解决这个问题。

Can someone please propose a plan of action for me?

有人可以为我提出一个行动计划吗？

Thanks

谢谢

From the Server.xml:

从 Server.xml：

    <Server port="-1" shutdown="__SHUTDOWN__">

From Catalina.out:

来自 Catalina.out：

Apr 28, 2014 5:34:50 PM org.apache.tomcat.util.digester.SetPropertiesRule begin
WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Valve} Setting property 'remoteIpProxiesHeader' to 'x-forwarded-by' did not find a matching property.
Apr 28, 2014 5:34:50 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 5:34:50 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 5:34:50 PM org.apache.catalina.startup.Catalina load
INFO: Initialization processed in 3247 ms
Apr 28, 2014 5:34:50 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service Catalina
Apr 28, 2014 5:34:50 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.53
Apr 28, 2014 5:34:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 5:34:59 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 5:34:59 PM org.apache.catalina.startup.Catalina start
INFO: Server startup in 8278 ms
Apr 28, 2014 5:41:53 PM org.apache.jasper.compiler.TldLocationsCache tldScanJar
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 10:32:32 PM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service Catalina
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-apr-8009"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["http-apr-8080"]
Apr 28, 2014 10:32:32 PM org.apache.coyote.AbstractProtocol destroy
INFO: Destroying ProtocolHandler ["ajp-apr-8009"]
May 05, 2014 8:10:32 PM org.apache.catalina.core.AprLifecycleListener init
INFO: Loaded APR based Apache Tomcat Native library 1.1.29 using APR version 1.3.9.
May 05, 2014 8:10:32 PM org.apache.catalina.core.AprLifecycleListener init
INFO: APR capabilities: IPv6 [true], sendfile [true], accept filters [false], random [true].
May 05, 2014 8:10:32 PM org.apache.tomcat.util.digester.SetPropertiesRule begin

回答by Will Hartung

Something is telling Tomcat to shut down.

有东西告诉 Tomcat 关闭。

Tomcat does not gracefully shut down when it's "in trouble" (out of memory, or anything else). Linux does not gracefully shut down processes when it's low on memory, it stops them hard with a kill -9, which leave no trace in the Tomcat log.

Tomcat 在“遇到麻烦”（内存不足或其他任何情况）时不会正常关闭。Linux 在内存不足时不会正常关闭进程，它会使用 kill -9 来硬停止它们，这在 Tomcat 日志中不会留下任何痕迹。

• kill -15will shut down tomcat gracefully, via the shut down hook. (The answer previously said kill -3here, which is simply causing a thread dump.)

• An internal exit() will shut it down gracefully as well, again via the shut down hook.

• It can be shut down via JMX, and, of course the shut down command from Catalina.

• kill -15将通过关闭钩子优雅地关闭 tomcat。（前面说的答案kill -3here，这只是导致线程转储。）

• 内部 exit() 也会优雅地关闭它，再次通过关闭钩子。

• 它可以通过 JMX 关闭，当然还有来自 Catalina 的关闭命令。

So, someone, somewhere, is telling Tomcat to do this. This isn't some unknown, internal "problem". Those kind of things just kill tomcat outright, they don't ask politely.

所以，某处有人告诉 Tomcat 这样做。这不是一些未知的内部“问题”。那种东西直接杀了tomcat，他们不客气地问。

One thing you might try is to enable DEBUG for logging, to see what chatter you get. If you get too much, you can try looking specifically at org.apache.catalina.core.StandardContext. This may well not give you more than it already being logged, but it may tell you something.

您可以尝试的一件事是启用 DEBUG 进行日志记录，以查看您收到的聊天内容。如果你得到太多，你可以尝试专门查看org.apache.catalina.core.StandardContext. 这可能不会为您提供比已经记录的更多的信息，但它可能会告诉您一些信息。

After that, if so motivated, and it continues, I'd suggest building Tomcat yourself, and interjecting some stack trace dumps.

在那之后，如果有动力的话，我会建议自己构建 Tomcat，并插入一些堆栈跟踪转储。

For example, in org.apache.coyote.AbstractProtcol, you can add:

例如，在 中org.apache.coyote.AbstractProtcol，您可以添加：

Exception e = new Exception("Shut down trace");
getLog().info("Shutting down protocol", e);

to the stop()method.

到stop()方法。

That will give you a solid stack trace to learn from whence this came. Tomcat isn't really a nasty event storm of just random messages floating around. The shut down is pretty synchronous. So, that stack trace will give you a solid lead as to who started the party. Then you can look from there to see how that could have happened.

这将为您提供可靠的堆栈跟踪，以了解这是从哪里来的。Tomcat 并不是一个令人讨厌的事件风暴，它只是四处飘散的随机消息。关闭非常同步。因此，该堆栈跟踪将为您提供有关谁发起派对的可靠线索。然后你可以从那里看，看看这是怎么发生的。

There's a bunch of interfaces and abstractions in Tomcat, but really only a few implementations, so it's not impenetrable. And having a smoking stack trace will help immensely in pinning it down.

Tomcat 中有一堆接口和抽象，但实际上只有几个实现，因此并非不可理解。并且有一个吸烟堆栈跟踪将极大地帮助确定它。

回答by Gerard H. Pille

How is your Tomcat started? If there is a parent process, a signal sent to the parent could be the cause of a clean shutdown.

你的Tomcat是怎么启动的？如果有父进程，发送到父进程的信号可能是干净关闭的原因。

回答by e.thompsy

There is a known (and fixed bug) in Tomcat versions prior to v7.0.55:

v7.0.55 之前的 Tomcat 版本中存在一个已知（并已修复）的错误：

https://bz.apache.org/bugzilla/show_bug.cgi?id=56684

https://bz.apache.org/bugzilla/show_bug.cgi?id=56684

The thread is really good reading but I can summarize it for you. When you start Tomcat there is a wait state that starts waiting for the shutdown command. This is normal, but the bug in these versions is an implicit timeout on this wait of about 50 days. When the wait state timed out, the Tomcat would shutdown.

该线程非常适合阅读，但我可以为您总结一下。当您启动 Tomcat 时，有一个等待状态开始等待关闭命令。这是正常的，但这些版本中的错误是在大约 50 天的等待中隐式超时。当等待状态超时时，Tomcat 将关闭。

So there ARE cases of odd things happening in Tomcat and NO, you are not crazy. This was happening to a SOLR cluster that I maintained for more than a year before I found this bug and fixed it with an upgrade. No one ever noticed because it took so long to crash and with all that goes on it can be easy to brush past finding the root cause. The easiest thing is to just restart Tomcat and forget it ever happened.

所以在Tomcat中发生了奇怪的事情，不，你没有疯。这发生在我维护了一年多的 SOLR 集群上，然后我发现了这个错误并通过升级修复了它。没有人注意到，因为崩溃花了很长时间，而且随着所有事情的发生，很容易找到根本原因。最简单的方法是重新启动 Tomcat 并忘记它曾经发生过。

I hope this answer helps you. I see that you are/were running a version with this bug.

我希望这个答案对你有帮助。我看到您正在/正在运行带有此错误的版本。

Good luck!

祝你好运！

回答by leeyuiwah

A case in DC/OS

DC/OS 案例

Recently we had a similar incident. Our tomcatis running in a docker as a service in a DC/OSenvironment. The stderr just said tomcat was killed, and there was no useful traces in any log files of tomcat.

最近我们也发生了类似的事件。我们tomcat在 docker 作为DC/OS环境中的服务运行。stderr只是说tomcat被杀了，tomcat的任何日志文件中都没有有用的痕迹。

It turned out the reason was simply that we did not allocate enough memory to the service. More debugging suggestions are here, but the key technique we used is to check the kernel log:

原来原因很简单，我们没有为服务分配足够的内存。更多调试建议在这里，但我们使用的关键技术是检查内核日志：

# journalctl -f _TRANSPORT=kernel
...
Mar 29 15:09:09 host-17 kernel: Memory cgroup out of memory: Kill process 7935 (java) score 1021 or sacrifice child
Mar 29 15:09:09 host-17 kernel: Killed process 7906 (java) total-vm:11583840kB, anon-rss:1032860kB, file-rss:30924kB, shmem-rss:0kB
...

Once we found that out, we increased the memory allocation to the service in DC/OS and the problem was solved.

发现后，我们增加了对 DC/OS 中服务的内存分配，问题就解决了。

回答by qudongfang

How is your Tomcat started? If there is a parent process, a signal sent to the parent could be the cause of a clean shutdown. I suggest adding set -moption into start_script.sh.

你的Tomcat是怎么启动的？如果有父进程，发送到父进程的信号可能是干净关闭的原因。我建议将set -m选项添加到start_script.sh.