I’am getting following error in IS server.log: [ISS.0137.0010E] Scheduler: Resources unavailable: Rolling back due to scheduler thread throttle reached:400
Here are some facts:
we’re running IS 7.1.2 incl. all latest fixes
task scheduler has around 40 services scheduled
resources/maximum threads = 500
scheduler thread throttle = 80% (400 threads)
IS stats still shows around 330 system threads
IS still working fine but tasks in scheduler are getting less threads and stop working after a while.
Any idea, what’s going on and how to monitor the current amount of scheduler threads?
WHat’s your setting for:
Settings > Resources>Scheduler Thread Pool
Maximum Threads
make sure it’s more than the number of scheduled jobs.
also, check how many running thread when this happens. if you still have free memory when it’s happening, and you running out of thread, you can increase the max thread setting.
couple of questions, have you migrated your schedulers to database ? and what are the settings for your schedulers? I ma more interested in parameters where you mention to run the service invocation after completion. Reason I am asking the second question is because you have 40 services scheduled which are far less than your threads allotted. There might be the case where where your same schedulers are running at the same time in a parallel manner that is why you are running out of scheduler threads.
Offcourse you can always increase the thread but the solution varies depending on the environments and needs.
There is still enough memory and are still enough server threads available.
For me it looks like an internal counter of scheduler threads is increasing but not really using the threads anymore. And when exceeding the threshold in my case 400 threads the error message will turn up.
I’m looking for a tool/service to monitor the current number of scheduler threads.
Thanks in advance.
yep, the schedulers have been migrated to database.
All scheduled tasks are running as Simple Interval Tasks (interval 30 seconds).
I thought the schedulers cannot run the same time when enabling ‘repeat after completion’. So, there shouldn’t be a backlog of services in case a service takes longer than 30 sec to complete.
Another interesting issue, when setting the scheduler thread throttle to 200% (I agree , this is quite a weird idea as the IS should be in trouble after a while) nothing really happened. After a while the same error message turned up saying that 1000 scheduler threads were in use (right, 200% of maximum of 500 service threads).
Therefor I’m looking for a tool/service to monitor the current number of used scheduler threads. I need to find out why this number is constantly increasing…
Ok I wanna put your attention to few things which might help you resolve this issue:
-Check the maximum number of connection you have mentioned in your JDBC pool which you are using for scheduler, maybe they are not enough.
-Ask you DB Administrator the interval when they release the session from DB side. Because sometimes you service finishes fine but DB doesn’t release the session which might be causing the reserve the thread on IS level. Normally DB Admins runs a schedule service to release expired session or the sessions which are not in use.
-The JDBC connection which you are using does it have enough connections to satisfy the needs of schedulers and other adapter services?
-Did you ever had this problem when you were using the embedded database for scheduler? if not then just try to create this problem with embedded database because that can help you isolate the problem.
P.S. Was there any specific need to migrate schedulers to DB ? was it to fulfill clustering requirement or something because by migrating scheduler to DB you just added another potential failing point.
The maximum number of connections in my JDBC pool which I am using for scheduler = 300 (therefore shouldn’t be a problem).
I asked my DB Administrator about the interval when they released the session from DB side. I got the feeling he didn’t really understand what I was talking about … He didn’t know anything about releasing sessions…
Re your question using the embedded database for scheduler…
Well, I guess this was a brilliant idea
I changed JDBC pool ISInternal from DB Oracle to Embedded DB. It’s currently looking pretty good as the number of threads used by scheduled tasks is not increasing anymore.
The issue was fixed with IS_7.1.2_Core_Fix15 or IS_8.0_SP1_Core_Fix7
±–
PIE-8908
Integration Server displays a message indicating that the scheduler thread throttle has been met even though it has not.
When several scheduled tasks execute concurrently, after a time, Integration Server displays the following message:
[ISS.0137.0010E] Scheduler: Resources unavailable: Rolling back due to scheduler thread throttle reached
Integration Server throws this message even when the number of threads used for executing scheduled tasks does not exceed the number of threads in the server thread pool that can be used for scheduled tasks.
I need to warm up this thread again, as our client frequently sees this error message in their logs. In one-minute-intervals the error is thrown, though IS seems to be running normally.
As we cannot just install IS_7.1.2_Core_Fix15 for various reasons (currently they have IS_7.1.2_Core_Fix6 installed), I am seeking a solution how to stop IS from throwing the error except restarting it completely.
Is there a single component that can be disabled/enabled or reset in any way so that IS will stop throwing the error. Can the thread count forcibly be refreshed to its true value somehow?
