Problem Description:
The number of threads keeps on increasing and after a certain limit (around 1200 for my environment), the server fails to respond after that; You have to restart the IS instance to free up all the hanging system threads and start afresh.
Analysis:
This happens when you repeatedly call “enableConnection” to enable JDBC adapter connection for a database which is actually down.
Has anyone else has faced this issue and knows of the solution/fix?
I will be posting a test package to prove this issue (successfully replicated the issue in IS6.1 and IS6.1 SP2)
To prove the above, I have setup this package. Create a new JDBC Adapter connection and bring the database associated with this newly created test adapter down;
Change the adapter alias setting in the flow service “TestSetResourceState” before running the service.
As time goes by, you will see in the error logs that the number of System Threads are on the rise; Please note to stop the flow service, you have to start the associated database.
Also, this happens with either using set of enable/disable connection services or set of get/set ResourceState services.
There is a scheduled service on the IS which will be calling the enable connection once every week(irrespective of the status of the connection) and enabling the connection.
Do you think the system threads will be increased in this case too.
I think you need to test that; You can use the sample packge I provided to test it; Do note that 1 thread per invocation per week is not too much to get noticed.
anyways, which “scheduled service on the IS” you are taking about? is it custom service or in-built one?
smallByte,
does this mean that you face this issue in your env too??
Meanwhile, I have opened an SR with webMethods (4 weeks back); Lets see what their final response is. They are still trying to get all the facts/thread dumps etc.
what you can do, is to take a thread dump every day and check it with the following free tool (Samurai):
[url]Samurai
On Unix-Environments you will need graphic access to your system for this tool.
File to be monitored is nohup.out when the server is running as “nohup server.sh &”.
On Windows this is a bit more difficult to extract the Thread Dumps from JVM, especially if your IS is running as a Service.
The output of the tool shows you the threads and their states in the thread dumps. If you have multiple dumps in one file you can compare the states from dump to the other to check if certain threads are blocked or deadlocked.
Thanks Holger… its a neat nice tool for analysing the thread dumps… But since the issue is in one of the webMethods packages (WmArt) or in the core IS jars, its currently out of our purview.
I have opened an SR and sent them all the information requested. Hopefully we get a fix/patch soon.
But what I would be interested in knowing is if someone on this forum can replicate the issue. Did you try running the “TestSetResourceState Package” and the test case?