Server keeps getting hung by TN Job Mgr thread

, ,
1

I’ve hit a peculiar problem with an IS 4.6 servers in the last few days – a single Java thread has been hogging CPU (90% and above). Given enough time (2-3 days) our server becomes unresponsive and must be killed and restarted.

I obtained a few thread dumps of the problem thread. They show that the “Trading Network job manager” is the thread is hogging the CPU. The problem is that IS server does not have any TN tasks pending (delivery or otherwise).

Current Thread Details

"Integration Server Thread" (TID:0x45b537b8, sys_thread_t:0x52bc83c8, state:R, native ID:0x2f07b) prio=5 
    at java.lang.Thread.yield(Native Method) 
    at com.wm.app.tn.delivery.JobMgr.processJobs(JobMgr.java(Compiled Code)) 
    at com.wm.app.tn.delivery.JobMgr.run(JobMgr.java(Compiled Code)) 
    at java.lang.Thread.run(Thread.java:498)

Also, the thread dumps show some interesting facts about Java monitor usage – TN seems to be expanding the monitor pool on demand, but free monitors seem to monotonically decrease over time.

Here is how the thread dumps show monitor usage:

bash# grep “Current number of free monitors:” javacore28007.10576*
javacore28007.1057628534.txt: Current number of free monitors: 64
javacore28007.1057632859.txt: Current number of free monitors: 55
javacore28007.1057633650.txt: Current number of free monitors: 55
javacore28007.1057637864.txt: Current number of free monitors: 46
javacore28007.1057643058.txt: Current number of free monitors: 29

I’m waiting for WM Support to help, but has anyone got any ideas or some across something similar? One recent change was installing IS_4-6_SP2_Fix113 on it.(I’ll try uninstalling it).

Best regards,
Sonam

2

Uninstalling FIX 113 didn’t fix the problem above, but it did fix this problem with Anonymous ACLs