java Java线程阻塞

Question

提问by Alexandre Boudreault

i have a problem with my java environement. I'm running Solr 1.3 (search engine) since more then a year now and suddenly i got alot of trouble with it. All my thread pool (250) got randomly blocked once or twice a day. I did not make any change on my solr application or my tomcat server.

我的java环境有问题。我运行 Solr 1.3（搜索引擎）已经一年多了，突然间我遇到了很多麻烦。我所有的线程池（250）每天随机阻塞一两次。我没有对我的 solr 应用程序或我的 tomcat 服务器进行任何更改。

I'm running tomcat 5.5.25 and Solr 1.3. I got a thread dump when the system is totally overloaded :

我正在运行 tomcat 5.5.25 和 Solr 1.3。当系统完全过载时，我得到了一个线程转储：

igot like 240 thread like this one :

我得到了像这样的 240 个线程：

"http-8080-Processor1" daemon prio=10 tid=0x0000000000b2e000 nid=0x193 waiting for monitor entry [0x000000004066c000..0x000000004066cb20]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at java.util.logging.StreamHandler.publish(StreamHandler.java:174)
    - waiting to lock <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
    at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:88)
    at java.util.logging.Logger.log(Logger.java:472)
    at java.util.logging.Logger.doLog(Logger.java:494)
    at java.util.logging.Logger.log(Logger.java:517)
    at java.util.logging.Logger.info(Logger.java:1036)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
    at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:619)

We can see that this thread is blocked and waiting on : <0x00007fe37e72b340>

我们可以看到这个线程被阻塞并等待：<0x00007fe37e72b340>

The thread who actually own the <0x00007fe37e72b340> is this one :

实际拥有 <0x00007fe37e72b340> 的线程是这个：

"http-8080-Processor156" daemon prio=10 tid=0x0000000000df2000 nid=0x1e52 runnable [0x0000000044521000..0x0000000044521c20]
   java.lang.Thread.State: RUNNABLE
    at java.io.FileOutputStream.writeBytes(Native Method)
    at java.io.FileOutputStream.write(FileOutputStream.java:260)
    at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
    - locked <0x00007fe37e3abcd8> (a java.io.BufferedOutputStream)
    at java.io.PrintStream.write(PrintStream.java:430)
    - locked <0x00007fe37e3abca0> (a java.io.PrintStream)
    at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
    at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
    at sun.nio.cs.StreamEncoder.implFlush(StreamEncoder.java:276)
    at sun.nio.cs.StreamEncoder.flush(StreamEncoder.java:122)
    - locked <0x00007fe37e72cd90> (a java.io.OutputStreamWriter)
    at java.io.OutputStreamWriter.flush(OutputStreamWriter.java:212)
    at java.util.logging.StreamHandler.flush(StreamHandler.java:225)
    - locked <0x00007fe37e72b340> (a java.util.logging.ConsoleHandler)
    at java.util.logging.ConsoleHandler.publish(ConsoleHandler.java:89)
    at java.util.logging.Logger.log(Logger.java:472)
    at java.util.logging.Logger.doLog(Logger.java:494)
    at java.util.logging.Logger.log(Logger.java:517)
    at java.util.logging.Logger.info(Logger.java:1036)
    at org.apache.solr.core.SolrCore.execute(SolrCore.java:1212)
    at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:303)
    at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:232)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
    at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:874)
    at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
    at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
    at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
    at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:689)
    at java.lang.Thread.run(Thread.java:619)

and this is the last part of my thread dump :

这是我的线程转储的最后一部分：

"ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon prio=10 tid=0x00007f6510349800 nid=0xbff waiting on condition [0x0000000041d8d000..0x0000000041d8dd20]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1548)
    at java.lang.Thread.run(Thread.java:619)

"pool-1-thread-1" prio=10 tid=0x0000000000c26400 nid=0xbfe waiting on condition [0x000000004200e000..0x000000004200eca0]
   java.lang.Thread.State: WAITING (parking)
    at sun.misc.Unsafe.park(Native Method)
    - parking to wait for  <0x00007f651b275510> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
    at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
    at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1925)
    at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:358)
    at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:946)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:906)
    at java.lang.Thread.run(Thread.java:619)

"Low Memory Detector" daemon prio=10 tid=0x00007f6510004400 nid=0xbfa runnable [0x0000000000000000..0x0000000000000000]
   java.lang.Thread.State: RUNNABLE

"CompilerThread1" daemon prio=10 tid=0x00007f6510001000 nid=0xbf9 waiting on condition [0x0000000000000000..0x0000000040d5e340]
   java.lang.Thread.State: RUNNABLE

"CompilerThread0" daemon prio=10 tid=0x00000000006bc400 nid=0xbf8 waiting on condition [0x0000000000000000..0x0000000040c5d2d0]
   java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00000000006bb000 nid=0xbf7 runnable [0x0000000000000000..0x0000000040b5da30]
   java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x0000000000690c00 nid=0xbf6 in Object.wait() [0x000000004065e000..0x000000004065ed20]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
    - locked <0x00007f651aa10258> (a java.lang.ref.ReferenceQueue$Lock)
    at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
    at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x000000000068f400 nid=0xbf5 in Object.wait() [0x000000004055d000..0x000000004055dca0]
   java.lang.Thread.State: WAITING (on object monitor)
    at java.lang.Object.wait(Native Method)
    - waiting on <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)
    at java.lang.Object.wait(Object.java:485)
    at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
    - locked <0x00007f651aa10338> (a java.lang.ref.Reference$Lock)

"main" prio=10 tid=0x0000000000622400 nid=0xbeb runnable [0x0000000000000000..0x00007fff69fcbba0]
   java.lang.Thread.State: RUNNABLE

"VM Thread" prio=10 tid=0x000000000068a000 nid=0xbf4 runnable 

"GC task thread#0 (ParallelGC)" prio=10 tid=0x000000000062cc00 nid=0xbec runnable 

"GC task thread#1 (ParallelGC)" prio=10 tid=0x000000000062e000 nid=0xbed runnable 

"GC task thread#2 (ParallelGC)" prio=10 tid=0x000000000062f400 nid=0xbee runnable 

"GC task thread#3 (ParallelGC)" prio=10 tid=0x0000000000630400 nid=0xbef runnable 

"GC task thread#4 (ParallelGC)" prio=10 tid=0x0000000000631800 nid=0xbf0 runnable 

"GC task thread#5 (ParallelGC)" prio=10 tid=0x0000000000632c00 nid=0xbf1 runnable 

"GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000000634000 nid=0xbf2 runnable 

"GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000000635400 nid=0xbf3 runnable 

"VM Periodic Task Thread" prio=10 tid=0x00007f6510006800 nid=0xbfb waiting on condition 

JNI global references: 1201

I know this is not a thread deadlock prob since one thread is actually runnning with all the ressource every other thread want.

我知道这不是线程死锁问题，因为一个线程实际上正在运行所有其他线程想要的所有资源。

Anyone got an idea of what can cause this prob ?

任何人都知道是什么导致了这个问题？

Answer 1

回答by BenoitParis

All your Threads are logging things. They all need to write on the disk from time to time. Everytime one of your 240 threads hits a logging line there will be disk access issues.

你所有的线程都在记录东西。它们都需要时不时地写入磁盘。每次您的 240 个线程中的一个遇到日志记录行时，都会出现磁盘访问问题。

It baffles me that the Thread having the lock is in the RUNNABLE state.

让我感到困惑的是，拥有锁的线程处于 RUNNABLE 状态。

I think it might be waiting for some external ressource to be released (like disk access for example)

我认为它可能正在等待释放一些外部资源（例如磁盘访问）

Are you running low on disk space? Have you recently changed something in your storage system?

您的磁盘空间是否不足？您最近是否更改了存储系统中的某些内容？

Answer 2

回答by David Newcomb

If you are running under Windows and the java application starts a console, be careful not to click in the DOS box. Window's crappy mark and copy "feature" blocks output to the ConsoleHandler. So any logger trying to write to the screen will block. Writing to the Console is done in a native call and so the java thread will appear to be in a RUNNING state when in fact it is blocked, it's just there's no way to feed that blocked status back to the application (because you are in native space).

如果您在 Windows 下运行并且 Java 应用程序启动控制台，请注意不要单击 DOS 框。Window 的蹩脚标记和复制“功能”块输出到 ConsoleHandler。所以任何试图写入屏幕的记录器都会被阻塞。写入控制台是在本机调用中完成的，因此 java 线程看起来处于 RUNNING 状态，而实际上它被阻塞了，只是没有办法将阻塞状态反馈给应用程序（因为你在本机空间）。

If the application is blocked (you have clicked in the DOS box) press escape to continue.

如果应用程序被阻止（您已在 DOS 框中单击），请按 Esc 继续。

Answer 3

回答by dstibbe

According to your log, the issue concerns the use of java.util.logging.ConsoleHandler.

根据您的日志，问题与 java.util.logging.ConsoleHandler 的使用有关。

First try to disable the console handler by removing it from the 'handlers' and '.handlers' list in '${TOMCAT_HOME}/conf/logging.properties'. See whether the problem still occurs.

首先尝试通过从“${TOMCAT_HOME}/conf/logging.properties”中的“handlers”和“.handlers”列表中删除控制台处理程序来禁用它。查看问题是否仍然存在。

If that helps, than it is definitely an issue with the output of the ConsoleHandler. Try checking whether there is are issues concerning the 'catalina.out' file. This is the file to where tomcat redirects its console output.

如果这有帮助，那肯定是 ConsoleHandler 输出的问题。尝试检查“catalina.out”文件是否存在问题。这是 tomcat 将其控制台输出重定向到的文件。

Answer 4

回答by Tom Hawtin - tackline

Flushing after each log record is going to expensive if you have very verbose logs.

如果您有非常冗长的日志，则在每个日志记录之后刷新会很昂贵。

A quality fix would be to clean up the logging, probably based around auditing.

质量修复是清理日志记录，可能基于审计。

As a quick fix, override StreamHandler.flushor OutputStream.flushto not do so immediately. Only flush once every so often. Note, however that you could potentially lose logging data immediately before a crash if you do this.

作为快速修复，覆盖StreamHandler.flush或OutputStream.flush不立即执行此操作。每隔一段时间只冲洗一次。但是请注意，如果您这样做，您可能会在崩溃前立即丢失日志数据。

Answer 5

回答by Victor Sorokin

I never used java.util.logging, so I don't know whether my suggestion is useful, but netherless:
try to use different instance of java.util.logging.Logger, so not all 240 threads will be blocked on the same monitor
(it will help if different instances of Loggeruse different instances of java.util.logging.ConsoleHandler) .

我从来没有用过java.util.logging，所以我不知道我的建议是否有用，但无济于事：
尝试使用不同的实例java.util.logging.Logger，所以并非所有 240 个线程都会在同一个监视器上被阻塞
（如果不同的实例Logger使用不同的实例，这将有所帮助java.util.logging.ConsoleHandler)。

Answer 6

回答by Thierry-Dimitri Roy

It seems that the thread that own "0x00007fe37e72b340" is blocked at the IO level. Maybe a disk (raid?) issue?

似乎在 IO 级别阻塞了拥有“0x00007fe37e72b340”的线程。也许是磁盘（raid？）问题？

can you do a thread dump 5 minutes later the see if the same thread is still blocked?

你能在 5 分钟后做一个线程转储，看看同一个线程是否仍然被阻塞？

java Java线程阻塞

提问by Alexandre Boudreault

回答by BenoitParis

回答by David Newcomb

回答by dstibbe

回答by Tom Hawtin - tackline

回答by Victor Sorokin

回答by Thierry-Dimitri Roy

相关推荐

最近更新

标签

java Java线程阻塞

提问by Alexandre Boudreault

回答by BenoitParis

回答by David Newcomb

回答by dstibbe

回答by Tom Hawtin - tackline

回答by Victor Sorokin

回答by Thierry-Dimitri Roy

相关推荐

java 在浏览器中显示 Excel 工作表？

java c3p0：我如何配置 autoreconnect=true？

java WEB-INF 目录中的 JSF 文件，我如何访问它们？

java 如何计算（并指定）java进程允许的总内存空间？

相关推荐

最近更新

标签