Java Weblogic 和 Oracle：卡住 ExecuteThread

Weblogic and Oracle: Stuck ExecuteThread

提问by tomraithel

We are running SpringMVC with Hibernate on a clustered Weblogic environment. Our application worked well, until we run into database space problems. Due to that, our DB provider extended the database storage.

我们在集群 Weblogic 环境中运行带有 Hibernate 的 SpringMVC。我们的应用程序运行良好，直到我们遇到数据库空间问题。因此，我们的数据库提供程序扩展了数据库存储。

After that change the Weblogic system status is complaining about the thread pool "Thread pool has stucked threads".

更改后，Weblogic 系统状态抱怨线程池“线程池已卡住线程”。

The stack-trace in the log file looks as follows:

日志文件中的堆栈跟踪如下所示：

<Error> <WebLogicServer> <BEA-000337> <[STUCK] ExecuteThread: '12' for queue: 'weblogic.ke
rnel.Default (self-tuning)' has been busy for "641" seconds working on the request "weblogic.servlet.internal.ServletReq
uestImpl@17782ff3[
GET /default/default/applicationPreloader/load.do?_=1331887527968 HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 5.1; rv:10.0.2) Gecko/20100101 Firefox/10.0.2
Accept: */*
Accept-Language: en-gb,en;q=0.5
Accept-Encoding: gzip, deflate
x-requested-with: XMLHttpRequest
Referer: http://novintegweb01.eu.acme.net:9136/default/default/applicationPreloader.do
Cookie: JSESSIONID1=NhBdPv9Hx8Wj5lnr5vnGhLSbR2GGM1DPhqT2YGzWCGFvhN3WJWVH!700169023!1391220923
Connection: Keep-Alive
Proxy-Client-IP: 192.168.1.1
X-Forwarded-For: 192.168.1.1
X-WebLogic-KeepAliveSecs: 30
X-WebLogic-Request-ClusterInfo: true
x-weblogic-cluster-hash: G4kGT1kfuDBbtj5w//SYNt6hlVo

]", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
        java.net.SocketInputStream.socketRead0(Native Method)
        java.net.SocketInputStream.read(SocketInputStream.java:129)
        oracle.net.ns.Packet.receive(Packet.java:293)
        oracle.net.ns.DataPacket.receive(DataPacket.java:104)
        oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:260)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:185)
        oracle.net.ns.NetInputStream.read(NetInputStream.java:102)
        oracle.jdbc.driver.T4CSocketInputStreamWrapper.readNextPacket(T4CSocketInputStreamWrapper.java:124)
        oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:80)
        oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1136)
        oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1113)
        oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:288)
        oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:191)
        oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:523)
        oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:207)
        oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:1009)
        oracle.jdbc.driver.OraclePreparedStatement.executeForRowsWithTimeout(OraclePreparedStatement.java:10117)
        oracle.jdbc.driver.OraclePreparedStatement.executeBatch(OraclePreparedStatement.java:10221)
        oracle.jdbc.driver.OracleStatementWrapper.executeBatch(OracleStatementWrapper.java:216)
        weblogic.jdbc.wrapper.PreparedStatement.executeBatch(PreparedStatement.java:210)
        org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:48)
        org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:247)
        org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:93)
        org.hibernate.jdbc.AbstractBatcher.prepareStatement(AbstractBatcher.java:88)
        org.hibernate.jdbc.AbstractBatcher.prepareBatchStatement(AbstractBatcher.java:223)
        org.hibernate.persister.collection.AbstractCollectionPersister.recreate(AbstractCollectionPersister.java:1119)
        org.hibernate.action.CollectionRecreateAction.execute(CollectionRecreateAction.java:39)
        org.hibernate.engine.ActionQueue.execute(ActionQueue.java:279)
        org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:263)
        org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:171)
        org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:298)
        org.hibernate.event.def.DefaultFlushEventListener.onFlush(DefaultFlushEventListener.java:27)
        org.hibernate.impl.SessionImpl.flush(SessionImpl.java:1001)
        org.hibernate.impl.SessionImpl.managedFlush(SessionImpl.java:339)
        org.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:106)
        org.springframework.orm.hibernate3.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:656)
        org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:754)
        org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:723)
        org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:393)
        org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:120)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.dao.CertificationDaoImpl$$EnhancerByCGLIB$$bf3135a5.getById(<generated>)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDateFromWs(UserProfileSabaServiceImpl.java:436)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.determineTargetDate(UserProfileSabaServiceImpl.java:401)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.createCertificationRelationObject(UserProfileSabaServiceImpl.java:373)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getRequiredLearning(UserProfileSabaServiceImpl.java:334)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.extractUserProfile(UserProfileSabaServiceImpl.java:203)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:73)
        com.acme.superproject.integration.UserProfileSabaServiceImpl.getUserProfile(UserProfileSabaServiceImpl.java:114)
        com.acme.superproject.integration.UserProfileSabaServiceImpl$$FastClassByCGLIB$$e48c61c9.invoke(<generated>)
        net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
        sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.integration.UserProfileSabaServiceImpl$$EnhancerByCGLIB$d7b85df.getUserProfile(<generated>)
        com.acme.superproject.service.UserProfileServiceImpl.getUserProfile(UserProfileServiceImpl.java:26)
        com.acme.superproject.service.UserProfileServiceImpl$$FastClassByCGLIB$cfe815.invoke(<generated>)
        net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:149)
        org.springframework.aop.framework.Cglib2AopProxy$CglibMethodInvocation.invokeJoinpoint(Cglib2AopProxy.java:688)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
        com.acme.superproject.techarch.TracingAspect.trace(TracingAspect.java:31)
        sun.reflect.GeneratedMethodAccessor5382.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
        org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
        org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:89)
        org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:621)
        com.acme.superproject.service.UserProfileServiceImpl$$EnhancerByCGLIB$bcb212b.getUserProfile(<generated>)
        com.acme.superproject.web.session.SessionDataProvider.fetchUserProfile(SessionDataProvider.java:97)
        com.acme.superproject.web.session.SessionDataProvider.getUserProfile(SessionDataProvider.java:78)
        com.acme.superproject.web.session.SessionDataProvider.fetchUserRegistrationCollection(SessionDataProvider.java:155)
        com.acme.superproject.web.session.SessionDataProvider.getUserRegistrationCollection(SessionDataProvider.java:141)
        com.acme.superproject.web.ApplicationPreloaderController.loadAction(ApplicationPreloaderController.java:84)
        sun.reflect.GeneratedMethodAccessor6506.invoke(Unknown Source)
        sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        java.lang.reflect.Method.invoke(Method.java:597)
        org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:176)
        org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:426)
        org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:414)
        org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:790)
        org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:719)
        org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:644)
        org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:549)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
        javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        weblogic.servlet.internal.StubSecurityHelper$ServletServiceAction.run(StubSecurityHelper.java:227)
        weblogic.servlet.internal.StubSecurityHelper.invokeServlet(StubSecurityHelper.java:125)
        weblogic.servlet.internal.ServletStubImpl.execute(ServletStubImpl.java:300)
        weblogic.servlet.internal.TailFilter.doFilter(TailFilter.java:26)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:368)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:109)
        org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:83)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:97)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:105)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.authentication.preauth.AbstractPreAuthenticatedProcessingFilter.doFilter(AbstractPreAuthenticatedProcessingFilter.java:89)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:79)
        org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:380)
        org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:169)
        org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:237)
        org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:167)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
        org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:76)
        weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56)
        weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3715)
        weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3681)
        weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321)
        weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120)
        weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2277)
        weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2183)
        weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1454)
        weblogic.work.ExecuteThread.execute(ExecuteThread.java:207)
        weblogic.work.ExecuteThread.run(ExecuteThread.java:176)
>

We already have restarted Weblogic, the database server and made a new deployment. The problem is always the same: When we try to access the login page of the application it hangs up - as far as I can see while trying to perform a DB transaction.

我们已经重新启动了数据库服务器 Weblogic 并进行了新的部署。问题总是一样的：当我们尝试访问应用程序的登录页面时，它挂断了 - 就我在尝试执行数据库事务时所看到的而言。

In addition to that, when I try to connect to our database with "Oracle SQL Developer" we also get the error "ORA-00257: archiver error. Connect internal only, until freed." Unfortunately we have no SYSDBA rights (because managed by an external provider).

除此之外，当我尝试使用“Oracle SQL Developer”连接到我们的数据库时，我们还会收到错误“ORA-00257：存档程序错误。仅在内部连接，直到被释放。” 不幸的是，我们没有 SYSDBA 权限（因为由外部提供商管理）。

My question are:

我的问题是：

• Has anyone of you had a similar problem before?
• Do you think the root cause of such problems is more on application side, or more on DB side?
• How can I at least get rid of the stucked threads?

• 你们中有人遇到过类似的问题吗？
• 您认为这些问题的根本原因更多是在应用程序端，还是更多在数据库端？
• 我怎样才能至少摆脱卡住的线程？

采纳答案by Jeff West

Chances are that the database storage is full, or that the archiver cannot move or create new archive logs. This is almost definitely an issue at the database level and not the app server level. Since you are getting this message that means you already have a connection to the database, so it is not sitting and waiting for a connection.

可能是数据库存储已满，或者归档程序无法移动或创建新的归档日志。这几乎肯定是数据库级别的问题，而不是应用服务器级别的问题。由于您收到此消息，这意味着您已经与数据库建立了连接，因此它不会坐等连接。

Stuck Threads in WebLogic don't always need to be 'fixed' and are not always a problem. The purpose of the stuck thread mechanism is to notify you when a thread is taking an abnormally long time to do something (like access the database). The stuck thread notification is just that - a notification that something might not be right, and in this case it is serving that purpose.

WebLogic 中的卡住线程并不总是需要“修复”，也并不总是一个问题。卡住线程机制的目的是在线程花费异常长时间来执行某事（例如访问数据库）时通知您。卡住的线程通知就是这样 - 通知某事可能不正确，在这种情况下，它正在服务于该目的。

If you have threads that routinely take 20 minutes to do something but want to be notified if it takes longer than 25 minutes, then have to increase the Stuck Thread Limitto 25 minutes and get the notification at that point then you must put this limit from 600 seconds to 1500 seconds (25*60).

如果您的线程通常需要 20 分钟来做某事，但如果需要超过 25 分钟，则希望收到通知，则必须将时间增加到Stuck Thread Limit25 分钟并在那时收到通知，那么您必须将此限制从 600 秒设置为1500 秒（25*60）。

You can edit stuck thread limit in weblogic admin base_domain > Environment > Servers > Admin Server > Configuration/Tuning.

您可以在 weblogic admin base_domain > Environment > Servers > Admin Server > Configuration/Tuning 中编辑卡住的线程限制。

refer below link : https://docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html

请参阅以下链接：https: //docs.oracle.com/cd/E19316-01/820-3745/ghudn/index.html

回答by Drona

This seems to be an issue with the database. The threads are stuck on the DB. The performance of your seems to have degraded. I would suggest to refer the case to your DBA for futher investigation. Also, try recreating your indexes on the tables being accessed.

这似乎是数据库的问题。线程卡在数据库上。你的性能似乎下降了。我建议将此案例提交给您的 DBA 进行进一步调查。此外，尝试在正在访问的表上重新创建索引。

回答by James Jithin

We have faced this issue when the Oracle DB Server was on a VirtualBox, in a developer environment. Pointing out the issue was bit difficult. We noticed this when the ThreadPooldid not have any more threads to handle the requests and the requests were rejected.

当 Oracle DB 服务器VirtualBox在开发人员环境中时，我们遇到了这个问题。指出这个问题有点困难。当ThreadPool没有更多线程来处理请求并且请求被拒绝时，我们注意到了这一点。