We are running 8.2 version on Unix box.
for the past few days schedulers were in running state for very long period of time.
once restarted everything went normal.
On analysing the thread dumps at the time of issue we found out following is causing the scheduler threads into waiting state.
Schedulers are in waiting for monitor entry as the following TriggerTask as a lock on the resource.
“TriggerTask:4:FXrLXXX.SubUpdxxxxxxxxxx.trigger:subXXXXXXXXXs” prio=7 tid=6000000003abf800 nid=107 lwp_id=991486 in Object.wait() [87fffffec76ff000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <87fffffef3866098> (a com.wm.driver.data.fs.FSData$Hash$Ent)
at com.wm.driver.data.fs.FSData$Hash$Ent.getEntry(FSData.java:3733)
- locked <87fffffef3866098> (a com.wm.driver.data.fs.FSData$Hash$Ent)
at com.wm.driver.data.fs.FSData$Hash.get(FSData.java:3653)
at com.wm.driver.data.fs.FSData.getEntry(FSData.java:2085)
at com.wm.driver.data.fs.FSDEntry.getReferencedEntry(FSDEntry.java:455)
at com.wm.driver.data.fs.FSDEntry.getIntRef3(FSDEntry.java:461)
at com.wm.driver.data.fs.FSDEntry.getNext(FSDEntry.java:355)
at com.wm.driver.data.fs.FSData.getNullEntry(FSData.java:2204)
- locked <87fffffee197fae8> (a java.lang.Object)
at com.wm.driver.data.fs.FSData.allocDataBlock(FSData.java:2497)
- locked <87fffffee197fad8> (a java.lang.Object)
at com.wm.driver.data.fs.FSData.allocDataBlock(FSData.java:2452)
at com.wm.driver.data.fs.FSData.createDataChain(FSData.java:2425)
at com.wm.driver.data.fs.FSDirCursor._encode(FSDirCursor.java:630)
at com.wm.driver.data.fs.FSDirCursor._setKey(FSDirCursor.java:625)
at com.wm.driver.data.fs.FSDirCursor._insertAfter(FSDirCursor.java:747)
- locked <87fffffee1a74b30> (a com.wm.driver.data.fs.FSDEntry$FSDEntryRef)
at com.wm.driver.data.fs.FSDirCursor.insertAfter(FSDirCursor.java:483)
at com.wm.util.data.TxnData$Element.backingCommit(TxnData.java:2295)
at com.wm.util.data.TxnData$Txn.commitTXN(TxnData.java:4219)
- locked <87fffffee1d4fd98> (a java.lang.Object)
at com.wm.util.data.TxnData$Txn.commitTXN(TxnData.java:4116)
at com.wm.app.store.impl.TSConsumer._persist(TSConsumer.java:460)
at com.wm.app.store.impl.TSConsumer.persist(TSConsumer.java:383)
at com.wm.app.b2b.server.dispatcher.PersistenceManager.persistToTriggerStores(PersistenceManager.java:409)
- locked <87fffffee0d61cc0> (a com.wm.app.store.impl.TSConsumer)
at com.wm.app.b2b.server.dispatcher.trigger.TriggerManager.deliverToTriggers(TriggerManager.java:522)
at com.wm.app.b2b.server.dispatcher.LocalProducer.deliverToTriggers(LocalProducer.java:33)
at com.wm.app.b2b.server.dispatcher.LocalProducer.persist(LocalProducer.java:51)
at com.wm.app.b2b.server.dispatcher.Dispatcher.publish(Dispatcher.java:266)
at com.wm.app.b2b.server.dispatcher.DispatchFacade.publish(DispatchFacade.java:223)
at com.wm.app.b2b.server.dispatcher.DispatchFacade.publish(DispatchFacade.java:186)
at wm.server.publish.publish(publish.java:304)
at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:443)
at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:643)
at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:41)
at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:243)
at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:51)
at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:276)
at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:30)
at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:363)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:547)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:386)
at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:234)
at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:94)
at com.wm.app.b2b.server.Service.doInvoke(Service.java:652)
at com.wm.app.b2b.server.Service.doInvoke(Service.java:540)
at pub.publish.publish(publish.java:146)
at sun.reflect.GeneratedMethodAccessor181.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at com.wm.app.b2b.server.JavaService.baseInvoke(JavaService.java:443)
at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:643)
at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:41)
at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:243)
at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:51)
at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:276)
at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:30)
at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:363)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:547)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:386)
at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:234)
at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:194)
at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:584)
at com.wm.lang.flow.FlowState.step(FlowState.java:444)
at com.wm.lang.flow.FlowState.invoke(FlowState.java:409)
at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1057)
at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:643)
at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:41)
at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:243)
at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:51)
at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:276)
at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:30)
at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:363)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:547)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:386)
at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:234)
at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:194)
at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:324)
at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:584)
at com.wm.lang.flow.FlowState.step(FlowState.java:444)
at com.wm.lang.flow.FlowState.invoke(FlowState.java:409)
at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1057)
at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:643)
at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:41)
at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:44)
at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:243)
at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:51)
at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:276)
at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:30)
at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:363)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:547)
at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:344)
at com.wm.app.b2b.server.dispatcher.trigger.Trigger.invokeService(Trigger.java:411)
at com.wm.app.b2b.server.dispatcher.trigger.Trigger.processMessage(Trigger.java:302)
at com.wm.app.b2b.server.dispatcher.trigger.DefaultTriggerTaskHelper.process(DefaultTriggerTaskHelper.java:253)
at com.wm.app.b2b.server.dispatcher.trigger.TriggerTask.run(TriggerTask.java:286)
at com.wm.util.pool.PooledThread.run(PooledThread.java:131)
- locked <87fffffee1da1d00> (a com.wm.app.b2b.server.TMPooledThread)
at java.lang.Thread.run(Thread.java:619)
Locked ownable synchronizers:
- None
Please suggest why it is happening.
Thanks
Regards
Hari