Too Many System Threads in WAITING state

,
1

Hello All,

I am currently facing issue that the system threads is getting increased at an alarming rate and too many threads are in WAITING state,

“NonStopCache [SoftwareAG.IS.Core.AMSDEV] Executor Thread-54 for ‘HTTP Handler 127.0.0.1’” Id=70855 in WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x3794b1fa> (a net.sf.ehcache.constructs.nonstop.NonstopThreadPool$Worker)
at java.lang.Object.wait(Object.java:503)
at net.sf.ehcache.constructs.nonstop.NonstopThreadPool$Worker.waitUntilTaskAvailable(NonstopThreadPool.java:276)
at net.sf.ehcache.constructs.nonstop.NonstopThreadPool$Worker.run(NonstopThreadPool.java:231)
at java.lang.Thread.run(Thread.java:724)

“Service Thread Pool” Id=70810 in WAITING
at java.lang.Object.wait(Native Method)
- waiting on <0x6153e7d5> (a com.wm.app.b2b.server.TMPooledThread)
at java.lang.Object.wait(Object.java:503)
at com.wm.util.pool.PooledThread.run(PooledThread.java:110)
at java.lang.Thread.run(Thread.java:724)

Please find the statistics below for the system threads.

Current	Peak	Limit

Total Sessions 4 16 -
Licensed Sessions 0 6 2147483647
Stateful Sessions 0 5 2147483647
Service Instances 5 37 -
Service Threads 3 9 -

System Threads 551 555 -

Below is the JVM memory settings for the IS,

set JAVA_MIN_MEM=2048M
set JAVA_MAX_MEM=2048M
set JAVA_MAX_PERM_SIZE=512M

Please find the server thread pool info also,

Available Threads 97 % (772 Threads)
Maximum Threads 800
Minimum Threads 50
Available Threads Warning Threshold 15 % (120 Threads)
Scheduler Thread Throttle 75 % (600 Threads)
Scheduler Current Threads 0

My question to the team is why the threads are staying in WAITING state instead of getting released.

Also we are seeing the below GC related message in the ehcache logs

ClientID[21]: Lock GC collected 4 garbage locks

Please help

Thanks and Regards
Abir Banerjee

2

See if there are any service which are running from a long time or if it is hung.

Restart the IS so that the waiting threads will be released.

3

Hello Mahesh,

We did a restart and again the threads are building up and going in WAITING state.

Thanks and regards’
Abir Banerjee

4

Attach error and server logs

5

What is your IS version? Is this a local installation setup?

And also you may have to analyse the JVM thread dump to see what is actually causing this.

6

Is this a prod env and also what did you see in the logs and service usage page any stale threads hung for processes?

Also based on your present load try to increase some more of resource threads to 1000+ and minimum to 100 and see the diff.

Maximum Threads 800
Minimum Threads 50

HTH,
RMG

1 Like
7

Recent versions of Ehcache (starting with 3.7, I think), which is used by Integration Server, require a lot of threads. As you’ve noticed, the threads are generally in a WAITING state. Several IS users have expressed concern about this, but it hasn’t caused any problems. These are system threads, so they do not come out of IS’s thread pool. A thread is a very lightweight object, so the presence of lots of threads do not necessarily indicate a problem.

To verify that this increase in threads is due to Ehcache, you will need to take a thread dump and look for threads with net.sf.ehcache classes in their stack frames. If you see lots of those, that confirms it.

Regards,
Michael

1 Like
8

We also noticed that there where way more threads on a 9.5 installation in comparison with a 7.1.3 installation
We asked software ag about it and this was there anser:

The thread count that you are seeing at the IS Admin Console → System Threads is currently active threads those are ready for use. It doesn’t mean that all of those threads are in use and processing Tasks. This count reflects top-level services only. Child services use the same thread that their top-level service uses. This count does not include services that are running from the wm.server. folder of the WmRoot package. May be your Integration Server has assigned 3 threads for it’s internal use. But I can see some of the System threads are single.

Even I have a clean install of Integration Server in my local desk and I can see around 200 threads are in active and ready for use.

For Example, If you can see that there are many “NonStopCache” threads. Regarding the NonStopCache threads, I can see these threads are being consumed by the following cache managers of ehcache.

SoftwareAG.IS.Core.TST_BCLUSTER1
SoftwareAG.IS.PE.TST_BCLUSTER1
SoftwareAG.IS.ART.TST_BCLUSTER1

these threads are not harmful and are being used by ehcache for some internal operations.

The Service Thread Pool size is not harmful and is expected, though there is no load at the time of taking the thread dump, I don’t think this will make any slowness in operation.

And also, LinkReaderThreads. Number of LinkReader, Broker Status, Broker Connection Keep-alive threads is as expected. So, in stable state conditions, whether system is idle or processing, we can expect few Broker Status threads, Keep-alive, and LinkReader threads. This number may go up or down if connections are broken, or a broker is not reachable, or new Broker JMS connections are getting created. It’s an expected behavior.

One small thing to keep in mind when you upgrate to a new version. You will have to add more memory due to the extra “WAITING” threads (every thread uses a amount of memory)
Sizing is not the same between old versions (before 9.x) and the current 9 versions.

9

Thanks a lot Jo for the inputs. This will definitely help.