Exception with Destination 'SessionDBDest' then IS was down

Cong_Ngo_Dinh · May 23, 2018, 3:21am

Hi Everyone,
My IS was running fine until this log appeared, then no transaction was wrote to log, it seem hang. I must restart IS then it works fine.
Any one know why that log appear ?
Thanks for any information.

2018-05-09 03:06:33 ICT [WmAuditingSC.destination.1122E] Destination ‘SessionDBDest’ caught unexpected Throwable logging audit event: [SoftwareAG][Oracle JDBC Driver]No more data available to read.
2018-05-09 03:06:33 ICT [WmAuditingSC.destination.1110E] Destination ‘SessionDBDest’ caught unexpected SQLException rolling-back transaction: [SoftwareAG][Oracle JDBC Driver]Object has been closed.
2018-05-09 03:06:33 ICT [WmAuditingSC.destination.1140E] Destination SessionDBDest caught an unexpected exception from listener com.wm.app.b2b.server.audit.AuditDBDestinationListener. We are proceeding with the close.
2018-05-09 03:06:33 ICT [WmAuditingSC.logger.3405E] In logger ‘Session Logger’, failed to log to destination; reason: Transient SQLException detected
2018-05-09 03:07:13 ICT [WmAuditingSC.destination.1122E] Destination ‘ErrorDBDest’ caught unexpected Throwable logging audit event: [SoftwareAG][Oracle JDBC Driver]The connect attempt failed because the server requires Oracle Advanced Security. To enable the driver to use OAS, please use the “dataIntegrityLevel” and/or “encryptionLevel” connect options.
2018-05-09 03:07:13 ICT [WmAuditingSC.logger.3405E] In logger ‘Error Logger’, failed to log to destination; reason: Transient SQLException detected
2018-05-09 03:07:22 ICT [ISS.0137.0011E] Scheduler: Exception: [SoftwareAG][Oracle JDBC Driver]Object has been closed.
2018-05-09 03:07:45 ICT [ISP.0090.0004I] Service makeTransfer - Request - MessageID(3eabaf47-d52f-4966-907c-b6d92cf55dd7) – This thing was hided.
2018-05-09 03:07:46 ICT [ISP.0090.0004I] Service makeTransfer - Response - MessageID(3eabaf47-d52f-4966-907c-b6d92cf55dd7) – This thing was hided.
2018-05-09 03:08:23 ICT [ISS.0137.0011E] Scheduler: Exception: [SoftwareAG][Oracle JDBC Driver]Object has been closed.
2018-05-09 03:08:23 ICT [WmAuditingSC.destination.1122E] Destination ‘ErrorDBDest’ caught unexpected Throwable logging audit event: [SoftwareAG][Oracle JDBC Driver]No more data available to read.
2018-05-09 03:08:23 ICT [WmAuditingSC.destination.1110E] Destination ‘ErrorDBDest’ caught unexpected SQLException rolling-back transaction: [SoftwareAG][Oracle JDBC Driver]Object has been closed.
2018-05-09 03:08:23 ICT [WmAuditingSC.destination.1140E] Destination ErrorDBDest caught an unexpected exception from listener com.wm.app.b2b.server.audit.AuditDBDestinationListener. We are proceeding with the close.
2018-05-09 03:08:23 ICT [WmAuditingSC.logger.3405E] In logger ‘Error Logger’, failed to log to destination; reason: Transient SQLException detected
2018-05-09 03:14:51 ICT [ISS.0137.0011E] Scheduler: Exception: [SoftwareAG][Oracle JDBC Driver]Object has been closed.
2018-05-09 05:41:16 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 14% available
2018-05-09 05:42:02 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 13% available
2018-05-09 05:45:22 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 12% available
2018-05-09 05:47:07 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 11% available
2018-05-09 05:48:07 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 10% available
2018-05-09 05:49:05 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 9% available
2018-05-09 05:49:59 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 8% available
2018-05-09 05:50:25 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 7% available
2018-05-09 05:51:10 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 6% available
2018-05-09 05:52:06 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 5% available
2018-05-09 05:56:47 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 4% available
2018-05-09 05:57:25 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 3% available
2018-05-09 05:58:09 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 2% available
2018-05-09 05:58:45 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 1% available
2018-05-09 05:59:26 ICT [ISS.0014.0020I] Available Thread Pool Warning Threshold Exceeded: 0% available

Yogesh_Bhandari · May 23, 2018, 8:39am

Hi Cong,

Your IS is going to hung state due to below issue.

Available Thread Pool Warning Threshold Exceeded: 0% available

You need to check your resources setting and increase the thread count in server thread pool based on your requirement.

Just navigate to Settings->Resources->Server Thread Pool.

Thanks,
Yogesh

Cong_Ngo_Dinh · May 23, 2018, 8:53am

Hi Yogesh Bhandari,
Yes, it’s hung because thread pool available is 0%, but that is not root cause. Request not much enough to use all threads, maybe problem come from exception related to DB.