Service cache causing locked threads

, ,
1

In IS 9.5, I am calling a service that has service caching enabled, properties Stateless=True and Cache results=True. During testing I performed a JVM thread dump and noticed many threads blocked.

Is there any way to get the service cache to work without locking? This is killing performance when many concurrent requests are being processed.

Running thread
…
at net.sf.ehcache.Cache.get(Cache.java:1590)
at net.sf.ehcache.Cache.get(Cache.java:1557)
at com.wm.app.b2b.server.ostore.EhcacheStore.load(EhcacheStore.java:104)
- locked <0x5ef5c1ca> (a com.wm.app.b2b.server.ostore.EhcacheStore)
at com.wm.app.b2b.server.ostore.CacheEntryList.access(CacheEntryList.java:75)
at com.wm.app.b2b.server.ostore.ServiceCacheImpl.get(ServiceCacheImpl.java:124)
- locked <0x32c9387b> (a com.wm.app.b2b.server.ostore.ServiceCacheImpl)
at com.wm.app.b2b.server.ostore.ServerCache.get(ServerCache.java:385)
at com.wm.app.b2b.server.CacheManager.invoke(CacheManager.java:196)
…

Blocked thread (there are many of these)
…
at com.wm.app.b2b.server.ostore.ServiceCacheImpl.get(ServiceCacheImpl.java:111)
- waiting on <0x32c9387b> (a com.wm.app.b2b.server.ostore.ServiceCacheImpl) owned by thread=“HTTP Handler 10.2.6.225” Id=28805
at com.wm.app.b2b.server.ostore.ServerCache.get(ServerCache.java:385)
at com.wm.app.b2b.server.CacheManager.invoke(CacheManager.java:196)
…

2

Never faced this before… I can suggest you can contact SAG global support with Java JVM thread dump. They can suggest you something.

Please let me know once you hear anything from them.

3

What IS memory you have set in the setenv.sh/bat file to handle the performance load for your IS to handle?

Also do you have any specific reason for that service cache enabled to true?

Also please touch base with SAG support with any core/thread heap dumps and they can suggest better settings.

HTH,
RMG

4

Thanks, I’ll open a support incident with SAG and include the thread dumps.

The heap size for the JVM is 8GB. There seems to be plenty of free memory available.

The service in which caching is enabled reads an XML file from disk and converts it to a document. To avoid disk i/o for each service call we enabled the service cache.

I will post an update once I hear back from SAG support.

5

Got it.

Any ways just a side node 6GB is the max you can give for any IS to handle JVM heap to my knowledge…

OK please update this thread back with findings.

HTH,
RMG

6

Hi Brian,

What is the value of watt.server.threadPool. Sometime we face issue in thread if we have low value. Check it out in server.cnf file.

7

Hi Lance,

The server thread pool is set to a max of 7500. This is higher than what we have set in production but we are in the process of performing stress tests and have been experimenting with different values. This just happens to be the current value in our test system.

SAG support has been engaged. I will post any updates.

Thanks,
Brian

8

You have more than required for server thread pool max 7500 interesting :smiley:
what is minimum set to and you can check it from the Resources page.

I am sure SAG support can assist you while peeking thru your specific environment and based on IS core/heap dumps you may have provided.

HTH,
RMG

9

The minimum thread pool setting is 10. After performing more tests I have noticed that not all thread dumps show blocked threads. I am hoping support can tell what’s going on.

10

Please raise the minimum to 50 or 100 also.

Still I believe your maximum is very very high number to me.

OK they might:

HTH,
RMG

11

Since I could not reproduce the locked threads in the thread dumps with enough consistency, webMethods support said that the threads are most likely released quickly and that it was normal behavior.

12

Have you got proper error handling in your cached service? Try to use debugLog service in your service catch sequence to see what error is being produced. Hard to see whats causing the problem. Can you explain what your service is doing?