JMS Trigger skipped processing few messages and stuck in Topic

, ,
1

Hi Everyone,

I have a weired Issue where i have published 300 messages to JMS Topic and few inbetween messages got stuck in UM Topic.

Example : Batch 1 to 150 processed, then 151 to 171 is stuck in topic and then rest 172 to 300 got processed successfully

Subscriber : JMS Trigger (Durable Subscriber - Topic)

I Could see messages were stuck in stuck in outstanding events and at the same time i could see below error in Nirvana.logs

[Cluster: UserThread-RequestManager-0] [com.pcbsys.foundation] - Socket Stream reach EOF
java.io.EOFException: Socket Stream reach EOF
at com.pcbsys.foundation.io.fEventInputStream.readComplete(fEventInputStream.java:398)
at com.pcbsys.foundation.io.fEventInputStream.readNative(fEventInputStream.java:419)
at com.pcbsys.foundation.io.fEventInputStream.readShort(fEventInputStream.java:220)
at com.pcbsys.nirvana.base.nHeader.readExternal(nHeader.java:925)
at com.pcbsys.nirvana.base.events.nBasePublishEvent.performRead(nBasePublishEvent.java:385)
at com.pcbsys.nirvana.base.events.nPublished.performRead(nPublished.java:161)
at com.pcbsys.nirvana.base.events.nPublishedJoin.performRead(nPublishedJoin.java:118)
at com.pcbsys.nirvana.base.events.nEvent.readExternal(nEvent.java:234)
at com.pcbsys.foundation.persist.fMultiFileStore.get(fMultiFileStore.java:263)
at com.pcbsys.foundation.persist.fMultiFileStore.get(fMultiFileStore.java:232)
at com.pcbsys.foundation.persist.fMultiFileStoreManager.getEvents(fMultiFileStoreManager.java:527)
at com.pcbsys.foundation.store.fStoreImpl.getEvents(fStoreImpl.java:114)
at com.pcbsys.nirvana.base.nChannelList.getEvents(nChannelList.java:289)
at com.pcbsys.nirvana.server.handler.nQueueEventHandler.getEvents(nQueueEventHandler.java:327)
at com.pcbsys.nirvana.server.handler.nQueueEventHandler.popEventFromQueue(nQueueEventHandler.java:244)
at com.pcbsys.nirvana.server.cluster.stateEngine.handler.nClusterQueuePopHandler.handleMasterUserRequest(nClusterQueuePopHandler.java:39)
at com.pcbsys.nirvana.server.cluster.stateEngine.nMasterState.userRequest(nMasterState.java:179)
at com.pcbsys.nirvana.server.cluster.stateEngine.cThreadedPipelineProcessor.processRequest(cThreadedPipelineProcessor.java:74)
at com.pcbsys.nirvana.server.cluster.stateEngine.cThreadedPipelineProcessor.run(cThreadedPipelineProcessor.java:114)
at com.pcbsys.foundation.threads.fThread.localRun(fThread.java:106)
at com.pcbsys.foundation.threads.hThread.run(hThread.java:102)
at java.lang.Thread.run(Thread.java:745)

Server Configuration details :
IS version : 9.12
IS Fixes :
IS_9.12_Core_Fix5
IS_9.12_SPM_Fix1

UM Version 9.12
UM Fix : 9.12 Fix 3

Restart of UM Fixed the issue but what may be cause of this issue and how to resolve it without restart of UM.

Regards
Deepak

um_outstanding events.jpg

2

That’s surprising, try installing the latest fixes for both IS and UM (Server and Client) and re-do your test.

If you still have the same issue, then we need to reach out to SAG.

3

Thanks Mahesh for the response.

Before even moving to latest fixes, i would like to understand the issue and the reproducible scenario , so i can confirm that issue would be fixed post applying fixes.

did anyone faced simillar issue?

Apart from error which i posted earlier in nirvana.log nothing else to zero in.

Regards
Deepak N

4

Did you try to reproduce the scenario by increasing the UM logs to TRACE, this will give you more details for your investigation.

5

Yes Mahesh, I have kept logging to trace and trying to reproduce.

I will post the updates as soon as i have one.

Regards
Deepak N