Incoming http client sessions not timing out

, ,
1

We are encountering a strange behaviour on our production IS. Lately the number of sessions are growing to 5000 and not being dropped by the IS. We have session timeout of 10 minutes and only http based incoming client connections. We have all our services as stateless.

Is there some other setting to be done so that these sessions are dropped.
Is there a way to call some service/code to clean up sessions which are unused.

Any clues highly appreciated.
Thx
Ritesh

2

In admin gui it shows negative session expires as -1200 secs. What does it signify…

3

You can add a garbage collection service running as a scheduled service every minute or so.
You collegue sushil has also raised this question here. Another method is to use the wm.server.admin:killSession service.
Good Luck!

4

You can also have your clients issue an explicit disconnect using /invoke/wm.server:disconnect service. Don’t forget that they will need to pass a cookie that was issued to them by the server.

5

wont an http disconnect suffice, and give a clue to webMethods to close the session.

Also I thought as per Session timeout settings webMethods wud drop the sessions after timeout of 10 mins. But strangely it shows session expires as -1300 secs and so on, on admin GUI. Is it a webMethods bug, or a way JVM works that only on next GC they wud be dropped.

6

Check the system tasks in the scheduler. Something could be hung in the Scheduler session preventing the Session Sweeper from running and removing the expired sessions.

7

Hi Alex,

Can you explain little more wm.server:disconnect.(cookie?)

8

Is there a way to kill hung System Tasks without rebooting IS ? I have an array of hung ‘Send Server Mail’ system tasks scheduled, which just keep on increasing everyday. SMTP and email settings in extended settings are all set to default (nil).

Thanks for help.

9

I met the same problem with Singh.
How to deal with it? and how to remove the expired sessions?

10

The symptom of “hanging System Tasks” is usually caused by some job (either in the User Tasks or in the System Tasks) that blocks. (I.e. runs into an endless loop, into a dead lock, etc.). Each time this job is started, it takes away one Thread from the System Tasks Thread-pool.
The default size of this pool is 5, I believe. So after such a blocking job has been started 5 times, the System Tasks pool is empty, and none of the System Tasks are executed anymore (including the “Session Sweeper”, so the expired Sessions are no longer removed).

Some possibilities for such a blocking job:

  1. If you schedule a Service, which uses the WmDB Services to access a database and there is a database problem. – Make sure to add enough error handling to the Service and in case of an Exception use pub.db:close with “$dbCloseConnection=true”. (This prevents that any subsequent executions of the Service get a stale DB connection.)
  2. Same for the pub.client.ftp Services. – There is a recent webMethods patch, which addresses problems with stale FTP connections.
  3. If you use Reverse Invoke, the “wm.server.security.revInvoke:keepAliveConnections” can cause a deadlock in the Scheduler as follows: assume that the Thread, in which an RI connection is started, notices that the connection is broken. It now tries to remove that connection from the RI connection pool. For this it first obtains the lock for the pool and then tries to obtain the lock for the connection. Now imagine that exactly at the same time the “keepAliveConnections” runs. It has acquired the lock for this connection and then pings it. The ping fails, so it tries to remove that connection from the pool and for this it needs the lock for the pool… So the result is: the Thread, in which the connection has been opened, has the lock for the pool and is now waiting for the lock for the connection, while the Scheduler Thread in which “keepAliveConnections” is running, has the lock for the connection and is waiting to obtain the lock for the pool! Then each minute a new “keepAliveConnections” is started, which runs right into this “traffic jam”, and very quickly the Scheduler is dead.
    I think in big RI scenarios this problem is one of the most common reason for the “hanging System Tasks” problem. – I brought this to the attention of webMethods, so there should be a fix available soon.
11

Hello all,

Can you please detail where the system tasks thread pool can be increased as we have the same issue?

In addition, can anyone explain how all the thread settings hang together. Basically which processes or tasks use the following thread settings:-

  1. server thread pool
  2. scheduler thread pool
  3. system tasks thread pool

Thanks in advance,

Steve.

12

A) In the web admin console, go to “Resources”->“Edit resource settings”.

B) server thread pool - all non-system services (ie. your own stuff)
scheduler thread pool - the scheduler & scheduled tasks
system task thread pool - the system threads

You also have the User Tasks Pool, TX Thread Pool and Cron Daemon Pool. The pools holds threads for what their names imply.

Chris

13

Hello Steve,
the System Tasks pool cannot be increased from the UI, but there is an undocumented server.cnf parameter, which does the Job:
shutdown the IS and then add
watt.server.cronMaxThreads=x
to the server.cnf file. The default value is 5.
Just for completeness: there’s also a parameter for the min value: watt.server.cronMinThreads, which defaults to 2.

But changing the System Tasks pool settings will only be “curing the symptoms instead of the disease”… You need to find the root cause of why your threads are hanging or dead-locking.

Regards, Ulrich