IS is unresponsive from Admin - what to do?

, ,
1

Our B2B IS freezes up after about two weeks in production. The admin is unreachable; the browser window is just blank as it waits. The log files do not contain any valuable information. What do you do to troubleshoot this freeze?

We are running on a RISC Unix box. Kill -3 has not produced a java core dump for some reason. Any ideas?

2

Investigate memory consumption as a possible issue. You can look at the stats log files for trending. If memory use keeps going up and never comes down, then one or more of your integrations is leaking resources.

3

1/ First of all, increase you log level (caution, it will consume more CPU power). You may found some interesting “No thread available” messages.

2/ Have a look on the messages issued by the JVM : we are launching our I.S. using nohup. Having a look on the nohup.log helped a lot to find out memory outage or JVM crashes.

3/ I’ve built a custom monitoring tracking several I.S. figures in flat file (Memory, # of threads, all JDBC pools, Broker …). A backIt helped a lot to find out why we ran out of resources.

4/ if not already done, don’t use the JVM provided by wM but prefere use the latest JVM provided by your operating system vendor.

Normal :
kill -3 = thread dump (the I.S. is not stopped)
kill = clean kill
kill -9 = abrutly kill the I.S. (use it only if the I.S is not responsive)

Bye

Laurent

4

It’s rather typical side effect of semi-loaded IS :wink:

  1. Use latest patches,
  2. Use technical restarts. One our Customer had to do technical restarts of about 40 IntegrationServers after average 3-4 days of running, and write his own monitoring console.
  3. Extend Java memory parameters:
JAVA_MIN_MEM=256M
JAVA_MAX_MEM=1408M
JAVA_MEMSET=-ms${JAVA_MIN_MEM} -mx${JAVA_MAX_MEM}
  1. Change GarbageCollector parameters to more aggressive. Try something like this on Sun Java 1.4.2:
JAVA_MEMSET="-Xms${JAVA_MIN_MEM} -Xmx${JAVA_MAX_MEM} -server -XX:TargetSurvivorRatio=95 -XX:SurvivorRatio=4 -XX:NewRatio=4 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled  -XX:+UseThreadPriorities -Xconcurrentio -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000"
  1. Moving to IBM Java. One our Customer relay on IBM Java, because it has better memory management out-of-the-box.
  2. Analyse core dump and logs - maybe it happens because of deadlocks, clash, memory leaks, session to long blocking and thread depletion, because of to complex synchronous logic in Your code.
  3. Monitor Your platform constantly or use external profiler on it,
  4. To kill multiple java processes (*nix old, native Java threading model) use fuser command on LOCKFILE file.

Easy way to break down IS instantly is to open >200 sessions (despite of max=1000 setted), another example is not proper use of adapters, recursive invocation, own transaction not proper management, and even in-IS-process ant task execution. I think, there seems to be a lot product-propriety “magic”, but really common sense and IS simple mechanic knowledge is enough.

5

Hi,

Recently I found out one more way of monitoring IS and MWS.
Add below JVM parameters (in case of jvm 1.5, for jvm 1.6 this is not required):

-Dcom.sun.management.jmxremote.port=
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

This will enable JVM monitoring capabilities to your JVM.

Utility called as Java VirtualVm is shipped with every JVM/JDK (1.5, 1.6 and up), .
Configure this to the server where IS / MWS is running and it will start monitoring heap usage, permspace, threads, classes etc…

You can even extract a threaddump without even restarting the server.

Regards,
Sumit

6

Yes, jvisualvm and jconsole are standardised tools since JDK 1.5, but these tools allow only monitoring of JVM and through JMX beans, but IntegrationServer not exposes any JMX beans like typical J2EE/JEE5 server does, so You can’t get a detailed information what’s IS really doing.
IS is build with use of part of Tomcat, and MWS on top on Jetty - MWS should expose some JMX beans - I’m not sure right now.

Willy Introscope is commercial tool giving more inside view of IS, but IS becoming unstable having Willy Introscope agent installed (IBM JDK 1.4.2 at RHEL).

7

jvisualvm can be used to generate thread dumps without the need to add any JMX parameter to the JVM, and yet this infotmation is usefull to understand what IS does, check deadlocks and so on…

introscope and other profilers causes problems.

Check your JVM is not running with -Xrs argument, which prevents signals.

Are you using IBM JVM? if so check the values for env variable IBM_JAVACORE_OUTOFMEMORY

8

Hi all and Happy New Year :wink:,

I got from a webMethod’s consultant a JMX plugin for MWS … but it crashed MWS.

I have already created a tool monitoring all wM resources (session, threads, JDBC, Brokers, …) but I’m missing all internal tread pools (i.e. mail, cron, …).

Is JMX able to expose all these internal figures ? a la “MIB” ?

Best regards,

Laurent

9

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.