Hello Techies,
I got a roblem with hung Threads on IS
Cancel and killing threads didn’t make any changes
and IS memory usage raises as the hung threads stall and raised up
is there any suggestion or recomended solutions to prevent these hung threads
coz, restart server is not the best way that we can do since this case happened in Production server.
Please help.
Thanks in advance.
Could you share more details:
Hi,
in addition to jerry´s questions:
Can you take and analyse some thread dumps on the JVM IS is running in?
This might help to identify which threads get stuck and why.
Regards,
Holger
Hello Jery,
we use IS 10.3 and there are several project running in these server
the hung thread specificly happned in one JDBC
the other JDBC didn’t run into the same problem (normally works as well)
we try to find service that couse this, but we have no clue at all
the service which running slow were randomly appears each time the hung happen
thx for the advice Tomsen,
ill try to analyze the thead dumps
but
is there any action that we can do, aside from restarting the IS server?
or maybe we can put timeout setting for the thread so that it wont be hung
Hi @Tegar_ASK
You can check the JDBC related information such as connectivity to the database - when the issue happens is the node reachable , is there a timeout after which instead of waiting response and being blocked may be will just terminate and return.
Thread dumps are of course the best way to go.
-NP
Hi @Tegar_ASK,
Since you have mentioned that the issue is happening for one specific JDBC adapter service, the next time this occurs you could run the service
pub.art.connection:getConnectionStatistics
this will give you an idea of the usage of the connection pool and its health, are there available connections or not.
You might also want to check with the developers if they are using a JDBC "Local Transaction"or "XA Transaction"as these too could result in long waits for locks to be released on the database (I personally stay away from using transactional connections unless absolutely required).
Also check with the development team what is it that they are are trying to do with the flow and if it has been tested in non-production environments with similar data volume.
You should also be able to identifiy the service that is taking long from the service usage page.
HTH
Regards
~Jerry
thx Nagendra,
is there any documentation that clearly describe how to read thread dumps
i’ll try to analyze our thread dumps
best regrads.
wow this is new things to me “pub.art.connection:getConnectionStatistics”
i’ll try to look up this service.
oh , is there any Transaction Type recomendation for APIs in which, the API traffic is massive and also db adapter using “stored procedure with signature” type?
Best regards
Hi,
if it goes for JDBC Connection you should check the min pool size configured and set this to 0 as this avoids having stale connections in pool.
Regards,
Holger
Hi @Tegar_ASK ,
The transaction type for the JDBC Adapter connection should reflect the type of operation its going to be used for please review the details of the “Transaction Types” along with "Transaction Isolation Level Settings"in the webMethods Adapter for JDBC Installation and User’s Guide 10.3 guide to get a better understanding of the transaction type you need to use. If you have no need for a "Local Transaction"connection I would strongly encourage that use a “No_Transaction” connection as much as possible.
HTH
~Jerry
Hello Holger,
honestly we allready set the min pool into zero
but its still happening
here some setting for the jdbc:
| Enable Connection Pooling | true |
|---|---|
| Minimum Pool Size | 0 |
| Maximum Pool Size | 300 |
| Pool Increment Size | 5 |
| Block Timeout (msec) | 30000 |
| Expire Timeout (msec) | 30000 |
| Startup Retry Count | 0 |
| Startup Backoff Timeout (sec) | 10 |
is there any suggestion?
best regards
thx Jerry for the advise
but we allready using the recomendation settup for jdbc (based on Guide)
and we still have the same issue…
| Connection Type | webMethods Adapter for JDBC Connection |
|---|---|
| Package Name | ******* |
| Transaction Type | NO_TRANSACTION |
|---|---|
| DataSource Class | oracle.jdbc.pool.OracleDataSource |
| Server Name | xxx.xxx.xxx.xxx |
| User | ******* |
| Password | ****** |
| Database Name | ******* |
| Port Number | ******* |
| Network Protocol | tcp |
| Other Properties | driverType=thin |
| Enable Connection Pooling | true |
|---|---|
| Minimum Pool Size | 0 |
| Maximum Pool Size | 300 |
| Pool Increment Size | 5 |
| Block Timeout (msec) | 30000 |
| Expire Timeout (msec) | 30000 |
| Startup Retry Count | 0 |
| Startup Backoff Timeout (sec) | 10 |
is there any misscofiguration on our jdbc?
BestRegards
Stability - Is your database connection and/or network stable? Increase the logging level for JDBC Adapter in the IS, and look in the server logs.
Locking - Thread dumps will show you if there is a locked thread, which is causing other requests to wait endlessly. These are Java thread dumps and there’s good content online to understand how to interpret them.
Driver Properties - Test these settings in the JDBC connection on a lower environment - driverType=thin;connectionProperties={oracle.net.CONNECT_TIMEOUT=10000,oracle.jdbc.ReadTimeout=60000}
Property Info -
KM
Can you share the config of the JDBC service? Is there a timeout configured for it?
I second that recommendation. And would also note that “No Transaction” type does not mean there is no transaction – it just means the adapter is not explicitly managing the transaction.
waow thank you Kasi for the details
Im gonna try it out
hope it gets better with your recomendation
I will post the result using this setup
as soon as possible
Best Regards
hello reamon
this is our jdbc timeout setting
but, it didnt work when there is a hung thread
i thougt that this setting will kill the hung thread,
but I realized that this timeout were works for killing the connection not the thread
cmiiw
@Tegar_ASK ,
Have you confirmed with your Oracle DBA’s that they are not seeing any database deadlocks on the tables that the Integration Server’s JDBC Adapter operates on?
Just curious as you have 300 connections.
Regards
~Jerry
well, they have confirmed that, at the time threads were hung
there was no deadlock on Oracle Session
it was really confusing what make the threads hung
and I wonder what should we do to these hung threads without entrupting the jvm
I ever read the same issue and there is a java script that could kill the thread immadiatelly but it will impact the jvm and gonna make another issue I guest
Best Regards