Adapter couldn't recover DB connection, instead it drained the thread pools and got stuck

engin_arlak · January 5, 2024, 8:58pm

Product/components used and version/fix level:

webMethods 10.3

JDBC_10.3_Fix18
IS_10.3_Core_Fix19

Detailed explanation of the problem:

We had a DB outage for around 40 minutes. During that time IntegrationServer tried to reestablish connection. When DB came back online, Integration Servers couldn’t recover the DB connection.

We observed that we had some stuck threads and our thread pool was completely full. It wasn’t able to recover the stuck threads and we couldn’t kill those threads manually either.

We tried disabling the adapter and re-enabling it again, but disable request stuck as well. This only made it worse because we tried to restart after realizing we can’t disable the adapter, then we tried reloading the package hosting the adapter connection and the service stuck, it didn’t help, it caused packages to be stuck as well; then we suspended the triggers and restarted the IS instance. Triggers were re-enabled automatically but the Adapter stayed disabled on boot. This caused messages to be consumed but not to be updated to DB. After enabling the adapter connection is established.

Error messages / full error message screenshot / log file:

Unfortunaltely, I don’t have all of the logs. We are trying to recreate the incident on test environment, will add more info once we have more info.

	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x7c56f8fd> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler *********" Id=1812
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)```

Question related to a free trial, or to a production (customer) instance?

commercial license, production environment

Questions are,
Did anyone had similar problems in the past and identified the root cause?
What would be the next problem identification step.

Any insight would be helpful. Thanks

Holger_von_Thomsen · January 8, 2024, 12:01pm

Hi Engin,

please consider updating all Pools to minSize=0, as the stale connections will be dropped the until the backend will be available again.

Additionally, you should consider enhancing your system maxThreads to 500 (worked well with 2GB JavaHeap for us in the past).

As the internal pools might be affected as well, they should be restarted after the outage is over via IS Admin UI.

Regards,
Holger

engin_arlak · January 9, 2024, 8:45pm

Hi Holger,

Connection pool minSize is already set to 0. The issue with the thread pool though, its really difficult to prove that its the culprit and adjusting that value is pretty much trial and error. There was a tuning baseline formula for that but I couldn’t find it anywhere.

We were able to reproduce the issue on test servers, but the problem didn’t persist after DB connection is back online. On production servers we had to restart the IS instances because it didn’t clear the stale threads after DB connection is restored. This is my only clue about the thread pool. It looked like there was a thread starvation problem. The question is, how do I prove that having too much thread pool can cause starvation? Is there an easier way then doing heavy load tests and changing thread pool while doing so?

edit: found the formula but its an old one.
https://empower.softwareag.com/sl24sec/SecuredServices/KCFullTextASP/viewing/view.asp?cstno=8103704&prdfamily=webMethods&KEY=439465-2319229&DSN=PIVOTAL&DST=SR

jahntech.cj · January 11, 2024, 10:59pm

Not on the actual topic, but the size of thread pools:

The question about a formula for the ideal size has been coming up for the last 20+ years. And of course things have changed due to improvements in the JVM, hardware, and yes also the OS. Contrary to widespread believe, the OS does have an impact on the behavior of the JVM, since at the end JVM threads need to be mapped to the OS. The latter can implement processes and threads (OS level, not JVM) in different ways.

The bad news is that there is no universal formula. And by definition there never will be one. The simple reason is that there are many, many factors which are unique to the implementation. How big is the amount of data per transaction? How many transactions in parallel? What OS? Is the code optimization to avoid CPU cache eviction (yes, you can do that in Java)? What about I/O for storage and networking and how that affects the number of file handles?

And we haven’t even talked yet about SLAs for latency and throughput. But for now I will stop here …

You can probably see that the number of factors outside the JVM itself is so big, that no universal formula can exist. There are some rough guidelines (max. JVM heap etc.) but they are only starting points. If anyone is interested in more details, please ask in a separate thread and tag me to ensure a response.

Thanks for triggering these thoughts!

engin_arlak · January 12, 2024, 3:26pm

Hi Christoph,

What I am looking for is not a formula but rather something as a starting point. AFAIK thread pool must be tuned by trial and error but testing high values is rather difficult. It never reaches to the max number, that’s good but when you have too many threads running in the background, doing context switches frequently will definitely degrade the performance. But the question here is, how do I tell if a value is too big for thread pool size? I can make an educated guess, but that won’t persuade anyone and it won’t be scientific deduction. Someone increased that number in the past, probably with a proper cause. Now if I reduce that pool size just because I feel like it especially when I don’t have any proof and if it breaks something in the production that will be a problem.

Holger_von_Thomsen · January 12, 2024, 4:02pm

Hi Engin,

can you try to get a thread dump for the JVM affected and check for the root cause for the blocked threads?

Thread Dump on Unix-like systems can be taken with a “kill -3 ” command and will be written to the wrapper.log in the profile-directory for this instance.

Regards,
Holger

engin_arlak · January 12, 2024, 4:22pm

Hi Holger,

Here is the last thread dump we generated when we reproduced the issue.

Fri Jan 05 16:51:07 MST 2024

"Event Manager Thread" Id=58268 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x7fca1f8b> (a com.wm.app.b2b.server.event.EMPooledThread)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:125)
	at java.lang.Thread.run(Thread.java:748)

"Event Manager Thread" Id=58267 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x2fb40c97> (a com.wm.app.b2b.server.event.EMPooledThread)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:125)
	at java.lang.Thread.run(Thread.java:748)

"Event Manager Thread" Id=58229 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x31eaf441> (a com.wm.app.b2b.server.event.EMPooledThread)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:125)
	at java.lang.Thread.run(Thread.java:748)

"Event Manager Thread" Id=58228 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x5ecea206> (a com.wm.app.b2b.server.event.EMPooledThread)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:125)
	at java.lang.Thread.run(Thread.java:748)

"webM RemoteHTTPEventHandler HTTPSocketMonitor" Id=58225 in RUNNABLE (running in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.net.SocketInputStream.read(SocketInputStream.java:224)
	at com.wm.app.b2b.server.comm.HTTPSocketMonitor.run(RemoteHTTPEventHandler.java:217)
	at java.lang.Thread.run(Thread.java:748)

"webM RemoteHTTPEventHandler HTTPSocketMonitor" Id=57258 in RUNNABLE (running in native)
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at java.net.SocketInputStream.read(SocketInputStream.java:224)
	at com.wm.app.b2b.server.comm.HTTPSocketMonitor.run(RemoteHTTPEventHandler.java:217)
	at java.lang.Thread.run(Thread.java:748)

"DailyFileAppenderTimer" Id=56570 in TIMED_WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x71d7c813> (a java.util.TaskQueue)
	at java.util.TimerThread.mainLoop(Timer.java:552)
	at java.util.TimerThread.run(Timer.java:505)

"Event Manager Thread" Id=56302 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x7e88b20a> (a com.wm.app.b2b.server.event.EMPooledThread)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:125)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56215 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x33d346e6> (a com.wm.util.synch.LatchedSemaphore)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.synch.LatchedSemaphore.semWait(LatchedSemaphore.java:56)
	at com.wm.util.synch.LatchedSemaphore.semWait(LatchedSemaphore.java:41)
	at com.wm.app.b2b.server.event.RemoteSubscriber.waitForEvent(RemoteSubscriber.java:55)
	at com.wm.app.b2b.server.comm.RemoteHTTPEventHandler.handleMessage(RemoteHTTPEventHandler.java:127)
	at com.wm.app.b2b.server.HTTPMessageHandler.process(HTTPMessageHandler.java:153)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x18a60966> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56197 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionManager.reset(WmConnectionManager.java:227)
	- waiting on <0x7cec559d> (a com.wm.app.b2b.server.jca.WmConnectionManager) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56186
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:401)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x7c0b97df> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x472872b6> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56196 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x6fedbe0c> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x46203161> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56195 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x6635f315> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x620c80a7> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56194 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x60935f34> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x27ffc17e> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56193 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x6e9b23cf> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x68149f03> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56192 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x128417ca> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x4b29d0d5> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56191 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x7c1b0724> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x30bb7323> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56190 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x02b318b9> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x0719df42> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56189 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x231fcaa2> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x48569789> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56188 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x523a28ea> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x1b901e20> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56187 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x46879d2a> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x3a11b0ac> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56186 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.resetPool(WmConnectionPool.java:753)
	- waiting on <0x58817dca> (a com.wm.app.b2b.server.jca.ConnectionPoolEntryDB) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.reset(WmConnectionManager.java:234)
	- locked <0x7cec559d> (a com.wm.app.b2b.server.jca.WmConnectionManager)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:401)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x396fa27c> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x2215ca6f> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56185 in WAITING
	at java.lang.Object.wait(Native Method)
	- waiting on <0x2b9a017c> (a com.wm.util.synch.LatchedSemaphore)
	at java.lang.Object.wait(Object.java:502)
	at com.wm.util.synch.LatchedSemaphore.semWait(LatchedSemaphore.java:56)
	at com.wm.util.synch.LatchedSemaphore.semWait(LatchedSemaphore.java:41)
	at com.wm.app.b2b.server.event.RemoteSubscriber.waitForEvent(RemoteSubscriber.java:55)
	at com.wm.app.b2b.server.comm.RemoteHTTPEventHandler.handleMessage(RemoteHTTPEventHandler.java:127)
	at com.wm.app.b2b.server.HTTPMessageHandler.process(HTTPMessageHandler.java:153)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x6f2232a6> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56184 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionManager.reset(WmConnectionManager.java:227)
	- waiting on <0x7cec559d> (a com.wm.app.b2b.server.jca.WmConnectionManager) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56186
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:401)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x27d0e0d2> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x74ecb6cf> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56183 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x0a251df6> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:122)
	at com.wm.app.b2b.server.WebServicesProcessImpl._process(WebServicesProcessImpl.java:1174)
	at com.wm.app.b2b.server.WebServicesProcessImpl.process(WebServicesProcessImpl.java:265)
	at com.wm.app.b2b.server.HTTPWebServicesHandler.process(HTTPWebServicesHandler.java:18)
	at com.wm.app.b2b.server.HTTPRootWebServicesHandler.process(HTTPRootWebServicesHandler.java:58)
	at com.wm.app.b2b.server.HTTPDispatch.handleRequest(HTTPDispatch.java:190)
	at com.wm.app.b2b.server.Dispatch.run(Dispatch.java:406)
	at com.wm.util.pool.PooledThread.run(PooledThread.java:134)
	- locked <0x71cf7c2f> (a com.wm.app.b2b.server.TMPooledThread)
	at java.lang.Thread.run(Thread.java:748)

"HTTP Handler xxx.xxx.xxx.xxx" Id=56182 in BLOCKED
	at com.wm.app.b2b.server.jca.WmConnectionPool.getConnection(WmConnectionPool.java:544)
	- waiting on <0x589b1d12> (a com.wm.app.b2b.server.jca.WmConnectionPool) owned by thread="HTTP Handler xxx.xxx.xxx.xxx" Id=56174
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateManagedConnection(WmConnectionManager.java:423)
	at com.wm.app.b2b.server.jca.WmConnectionManager.allocateConnection(WmConnectionManager.java:355)
	at com.wm.adk.cci.connection.WmConnectionFactory.getConnection(WmConnectionFactory.java:141)
	at com.wm.pkg.art.ns.ConnectionResource.createConnection(ConnectionResource.java:385)
	at com.wm.pkg.art.transaction.ConnectionState.getConnection(ConnectionState.java:197)
	- locked <0x5139c4b4> (a com.wm.pkg.art.transaction.ConnectionState)
	at com.wm.pkg.art.ns.AdapterServiceNode.getConnection(AdapterServiceNode.java:685)
	at com.wm.pkg.art.ns.AdapterServiceNode.invokeService(AdapterServiceNode.java:349)
	at com.wm.pkg.art.ns.ARTNSService.baseInvoke(ARTNSService.java:54)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.process(ValidateProcessor.java:49)
	at com.wm.app.b2b.server.invoke.PipelineProcessor.process(PipelineProcessor.java:171)
	at com.wm.app.b2b.server.ACLManager.process(ACLManager.java:326)
	at com.wm.app.b2b.server.invoke.DispatchProcessor.process(DispatchProcessor.java:34)
	at com.wm.app.b2b.server.AuditLogManager.process(AuditLogManager.java:401)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:638)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:447)
	at com.wm.app.b2b.server.invoke.InvokeManager.invoke(InvokeManager.java:405)
	at com.wm.app.b2b.server.ServiceManager.invoke(ServiceManager.java:253)
	at com.wm.app.b2b.server.BaseService.invoke(BaseService.java:236)
	at com.wm.lang.flow.FlowInvoke.invoke(FlowInvoke.java:267)
	at com.wm.lang.flow.FlowState.invokeNode(FlowState.java:668)
	at com.wm.lang.flow.FlowState.step(FlowState.java:534)
	at com.wm.lang.flow.FlowState.invoke(FlowState.java:501)
	at com.wm.app.b2b.server.FlowSvcImpl.baseInvoke(FlowSvcImpl.java:1150)
	at com.wm.app.b2b.server.invoke.InvokeManager.process(InvokeManager.java:768)
	at com.wm.app.b2b.server.util.tspace.ReservationProcessor.process(ReservationProcessor.java:46)
	at com.wm.app.b2b.server.invoke.StatisticsProcessor.process(StatisticsProcessor.java:54)
	at com.wm.app.b2b.server.invoke.ServiceCompletionImpl.process(ServiceCompletionImpl.java:250)
	at com.wm.app.b2b.server.invoke.ValidateProcessor.pro

What we couldn’t reproduce was after restoring the DB connection, Adapter connection getting stuck. In test environment, the threads that were stuck due to db connection issue was restored after db connection restored. It wasn’t the case in production. I am guessing we didn’t have as heavy load as prod environment that time and that was the reason they weren’t restored in prod environment (due to too frequent context switches). When we realised we had a production issue, we already had run out of thread pools.

Nagendra_Prasad · January 12, 2024, 4:43pm

Hi @engin_arlak
Looking at the thread dumps, it looks like there could be a deadlock happening here.

Quite a few threads are waiting on a resource held by Thread ID 56174 , as apparent from the lines containing “owned by thread=“HTTP Handler xxx.xxx.xxx.xxx” Id=56174”
However, there are no details of what the thread ID 56174 is , I’m assuming the thread dump pasted above was partial.
This thread is likely holding a lot of resources and waiting for another resource.

This does look like an instance where you need a support ticket to get this analyzed.

-NP

engin_arlak · January 12, 2024, 4:54pm

Unfortunately support is not an option atm. We thought we had extended maintenance agreement but it turns out we didn’t. It will take a while even if we want to purchase it now.

engin_arlak · January 12, 2024, 4:57pm

Thread dump is taken from IS admin UI. It seems like when DB connection is lost, then that thread is blocked and it will be in blocked state until the connection is established again. I don’t know what happens if those stuck threads fill up the thread pool entirely though. That was the case we had in production.

Nagendra_Prasad · January 13, 2024, 7:32am

Hi @engin_arlak
As you are already aware, one thread dump is not sufficient to make a meaningful assessment, however I found the high number of BLOCKED threads in the thread dump above a little unusual especially when they all are waiting on resources locked by a particular thread.

The reason that it looked like a partial thread dump is because it appears to be missing some lines at the end and the details of thread 56174 aren’t present.
I would suggest you take multiple ( say 3) thread dumps at constant intervals ( say 10 seconds). If the Integration Server is unresponsive you may have to use utilities like jstack or the methods that @Holger_von_Thomsen has mentioned in his post.

-NP

Holger_von_Thomsen · January 15, 2024, 10:50am

Hi,

there are two resources being blocked by Thread-ID 56174, which other threads are waiting for:
0x58817dca and 0x589b1d12
As all available blocked threads are referring the ConnectionPoolManager, you should check with your DBA if there is a Deadlock situation in the DB.

I used Samurai (plain java based UI in a jar-file) for viewing Thread Dumps logged to wrapper.log, but this will require some editing of this file (better take a copy before and edit the copy instead):

remove all lines which are not part of the thread dumps to be analazyed.
for the all remaining lines remove the “jvm 1|”-prefix
open the edited file inside UI (or just drag and drop there). After a short time the dumps will be analyzed and displayed in a table where you can see how the status of the threads change between different thread dumps when there are multiple thread dumps in the file.
You can then investigate the relations between the threads in more detail to detect the base blocking thread.

Having more JVM HeapSpace to allow for more threads in the ThreadPool will not solve the issue as long as you do not find the root cause for the base blocking thread.

Regards,
Holger

engin_arlak · January 16, 2024, 10:28pm

We know why we lost db connection, but it is not relevant to this problem. The problem here is not losing connection with DB. The problem is IS failed to restore connection to db and it also failed to clear blocked threads. Here in this thread dump, we blocked DB connection to reproduce the issue. When we reproduced, our test environment didn’t fail to clear blocked threads, and it was able to restore DB connection without an issue.

So, to summarize, when we had this issue in prod, it persisted days after DB connection restored(it didn’t until we restarted the IS but DB was up and running), until we restarted IS instances. This didn’t happen exactly the same when we produced but the thread dump and thread pool looked similar. This is the reason why I though it might be related to thread pool being too large. We didn’t have as heavy load as prod in test environment, all we did was to create the load manually.

The blocked threads are because of the DB adapter not having a connection. My coworker says that “In documentation its noted that when a connection is lost, the thread holding that connection is blocked hence this is expected”. I didn’t verify this myself though. Still what I expect is even if the thread pool is empty, it should still be able to restore the connection and clear the blocked threads one by one, even if this caused a thread starvation. It stayed blocked for days in production environment.

Sandeep_Deepala · January 24, 2024, 10:18pm

Iam expecting it to be a dead lock scenario, check with your DBA whether they are killing all Wm sessions before they restarted their database server. If DBA unable to help you, then try below settings in non-prod but I think it would be a bit risky.

watt.server.jca.connectionPool.threadInterrupt.waitTime
Specifies the maximum number of milliseconds that a thread can take while creating or closing a connection before the pool interrupter thread interrupts the thread. After the specified time elapses, the pool interrupter thread considers the thread to be blocked and interrupts it. There is no default value. The pool interrupter thread, which is a monitoring thread for the connection pool, only executes if this server configuration property has a value that is greater than zero.
You must restart Integration Server for changes to this parameter to take effect.

watt.server.jca.connectionPool.createConnection.interrupt.waitTime
Specifies the wait time interval in milliseconds, which elapses before Integration Server interrupts a connection creation thread that is in a wait state. The parameter does not require a default value.

watt.server.jca.connectionPool.threadInterrupter.sleepTime
Specifies the number of milliseconds the pool interrupter thread sleeps between sweeps for server threads that became blocked while creating or closing a connection. When the sleep time elapses, the pool interrupter thread, which is a monitor thread for the connection pool, checks for server threads that became blocked while attempting to create or close a connection from the connection pool. The default is 2000 milliseconds.

Important:
This parameter is applicable only when the underlying connection socket channels are interruptible.