Notification Thread Exhaustion

, ,
1

Yesterday I started getting alerts from the IS "
No threads available to run task {2#Notification- Cron Daemon Pool}".

I took a thread dump and analyzed and found the following:
There was a single notification thread (ThreadA) which had locked a monitor which almost all of the other notification threads were waiting on causing a block. This exhausted the notification thread pool and hence the error.

Before I started getting this alert, I did get an alert on the JDBC Adapter Connection ThreadA uses.
Adapter Runtime (Connection): Unable to get a connection to resource JDBCConnection.NotificationConnection:Notification.

My guess is when the connection issue happened this thread hung.

Here is my question:

  1. Is it correct understanding that a notification thread remains active from the time a trigger is fired till the service which follows completes? And if for some reason there is a break, the thread won’t get released?

  2. Is there any way we can build in the service to timeout after certain time or for the thread to be released after sometime?

  3. Also, has anybody implemented any way to kill threads in the IS JVM if you doubt a hang situation.

Nancy

2

I have done two things. Will see how it works for us.

No threads available to run task {2#Notification- Cron Daemon Pool}.

  1. I have set watt.adapter.JDBC.QueryTimeout to a value (60 seconds to start with)

  2. Increased the polling interval of the notification.

3

Hi Nancy,

I have to analyze JVM thread dump for one of my TN servers.
I have created the thread dump using Integration Server console 8.2 but don;t know how to analyze it.

Could you please help me with this.

Thanks,
Amit