How to Install Newer JVM for IS

, ,
21

Wayne,

The IBM JVM was originally shipped with earlier versions of the webm platform. Many people swear by it, but I switched because the original was a 1.3.0 version and I wanted a minimum of 1.3.1.

If you are running as a service, there is a lot more that you have to do because of some registry settings that are needed relating to the JVM.

It would be nice if webMethods gave the installer an option to choose the JVM (bundled or their own.)

I am more familiar with SUN because I have used it since day one over IBM and there is a TON of tuning/optimization information available from SUN.

INTEL may have some information relating to the IBM JVM on their site as I recall a benchmark whitepaper on this topic.

As far as System.gc();, tech support started giving me this “fix” when we had java.lang.OutOfMemory errors. I always kinda wondered what the server.cnf watt.server.cache.gcMins does because it seems like it does nothing at all. I run System.gc(); every five minutes.

Generally, my experience has been that the JVMs provided by the OEMs work better with their stuff. And, sometimes you have no other choice.

HTH

Ray

22

Thanks Ray.
You mentioned “there is a lot more that you have to do” if we are running the IS as a service.
We ARE running it as a service. The only thing we did was run the batch file installSvc.bat in <webMethods>\IntegrationServer\support\win32
Is there more to do ?

Wayne

23

If it worked for you, then you are truly blessed. For us, we had to perform a super-hack on the windows registry and dump some of the supporting files (Registry, FSData and some others). I didn’t work on this specific part of this project but I can scoop up the details if they are needed.

We had to do this because the installSvc.bat didn’t function properly for us.

Ray

24

Ray,

I think forcing a full GC using System.gc() may have been a necessity “back in the day” before newer generational garbage collection features were introduced (and improved) in the 1.3.1 and 1.4.x JDK’s.

If you read how garbage collection works on Sun’s Java tuning pages it appears that you could actually make performance worse by forcing a full GC when compared to a JVM that is using properly configured generational garbage collection. Note: different java vendors have different command line parameters that are used to tweak the JVM gc settings. I know of some internal webMethods performance tuning documents that discourage the practice of scheduling System.gc() for this reason.

Here’s an entry in the GC FAQ from [URL=“Oracle Java Technologies | Oracle”]Oracle Java Technologies | Oracle

31. Should I pool objects to help GC? Should I call System.gc() periodically?

The answer to these is No!

Pooling objects will cause them to live longer than necessary. We strongly advise against object pools.

Don’t call System.gc(). The system will make the determination of when it’s appropriate to do garbage collection and generally has the information necessary to do a much better job of initiating a garbage collection. If you are having problems with the garbage collection (pause times or frequency), consider adjusting the size of the generations.
[/quote]

For more on this related to the Sun 1.4.2 JDK see: [URL=“Oracle Java Technologies | Oracle”]Oracle Java Technologies | Oracle

BTW, Integration Server 6.1 (planned for a Jan. 2004 release) will support Java 1.4.1 or 1.4.2 depending on what platform you are using.

Mark

25

Hey Mark,

I agree with you in principal, but best practices doesn’t always meet your requirement. System.gc(); seems to be the castor oil of java. Tastes horrible but fixes all that ills.

I run mine every five minutes and it seems to do the trick. On the newer JVMs (of which I must confess I have never used) the parallel GC probably takes care of what the System.gc() does now.

I can say this with a straight face: When I am parsing VERY large files using node iterator, at the end of each repeat, I am known to stick in a call to System.gc() to flush out the dead stuff.

I have seen serious performance degradation by using System.gc() too much. It’s a touchy feely kinda thing.

Ray

26

I’m sure that you can OD on castor oil just like anything else.

Just keep in mind that if you write code (or schedule services) which force System.gc(), you may be negating the main advantage you would get from upgrading to a newer JDK in the future.

I hope to have a chance to do some JVM tuning soon and will report back if we find anything specific to webMethods.

Mark

27

This article ([url=“IBM Developer”]IBM Developer) on fine-tuning Java performance was referenced in the new “Advanced Troubleshooting Guide” recently published on Advantage.

The advanced troubleshooting guide can be downloaded by first logging into advantage in a new browser window and then clicking on the following link: [url=“http://advantage.webmethods.com/cgi-bin/advantage/logAndRedirect.jsp?w=0&s=3251635231&t=SOLUTION&targChanId=-536883716&oid=536919336&l=tslog200401.log”]http://advantage.webmethods.com/cgi-bin/advantage/logAndRedirect.jsp?w=0&s=3251635231&t=SOLUTION&targChanId=-536883716&oid=536919336&l=tslog200401.log[/url]

The Java performance tuning article identifies the concurrent and parallel processing capabilities of each phase of the garbage collection process by IBM JVM version.

IS 4.6 ships with IBM 1.3.0 and IS 6.1 will ship with IBM 1.3.1 for Windows. At the time of the 6.1 testing, IBM had not yet released their 1.4.x JVM.

Mark

28

Hmmm, this is a better URL for the advanced troubleshooting guide might be:


29

I just though I add my 2 cents here. Personally in my experiance I run into many memory problems using IS in different projects. WM support used to allways advise a first thing is do gc() call. However, I have never seen System.gc() calls fix or improve any of my memory or grabge collection issues in IS! On the other hand I have not seen it to do any harm iether. To me gc() call changed nothing had practically no effect. This is working on differnt JVM 1.1.8 1.3.0, 1.3.1 and 1.4.x from SUN or IBM alike on hardware by IBM AIX, SUN, HP or Windows and Linux.

What did help in most memory problem situations is Java parameter tuning, OS kernel tunning where possible and most importantly revising the code created by many people. Nearly in all cases memory problems were found inside the code rather than JVM. Once some minor java service was changed memory problems would go away. Its finding this small mistake usually a long task.

Take it as you will but that has been my experiance.

30

My first hand analysis of GC logs produced by a load test conducted last week showed “normal” gc behavior that responded as expected to changes in the size of the memory heap or to other vendor-specific GC parameters.

For example, in one test increasing the java heap size from 768Mb to 1024Mb decreased the frequency of garbage collections from 6 seconds to 12 seconds while slightly increasing the time required to complete the GC from 900ms to 1000ms.

If the verbose GC logs show issues related to either too frequent GC’s or excessively long GC pause times, these issues could be addressed successfully by iterative testing and tuning of the JVM parameters available from the JVM vendor.

In last week’s test an IS integration service was completely filling a 1.3Gb JVM heap every 12 seconds. The resulting GC’s completed in approximately 1 second under the IBM 1.3.0 JVM which uses a parallel mark phase, but single threaded sweep and compaction phases. I can see absolutely no need to introduce a System.gc() into this particular application.

Here’s my current take on IS performance tuning:

Mark

31

Hi all,

We’re conducting some load tests here also.
We are testing various configuration scenarios, changing the parameters watt.server.broker.producer.multiclient, watt.server.threadPool, watt.server.threadPoolMin and the garbage collection parameters when starting JVM, in a way to determine the best set of parameters according our application.
I couldn’t find enough information about how to configure a Solaris box in a way to best accommodate webMethods.
Can you guys give me some tips about OS configuration?
Any help will be fully appreciated.
BTW, we’re running wM6 over jvm 1.4.2 into Solaris SunOS 5.8.

Thanks,
Maldonado

32

Dead Horse Society,

Read this thread from top to bottom. After (and only after) reading the thread and the linked articles, post detailed, factual responses showing the “benefits” of scheduled System.gc() calls if you have any.

Mark

33

A long-time WM Users member asked what I meant by Dead Horse Society. I was referring to this post yesterday and assumed (incorrectly) that readers were familiar with the “beating a dead horse” idiom.

Mark

34

oh… oops…

I guess I was following one of these strategies.
Beating a Dead Horse

Cheers!

  • Saurabh.
35

Hi,
Can anyone let me know where to download IBM jvm 1.3.1? We are currently on IBM 1.3.0 version and planning to upgrade to 1.3.1 on Windows OS.
Thanks,
VL

36

It is included as part of the IBM WebSphere SDK for Web Services (WSDK) Version 5.1

This SDK contains the following components:

  • An embedded version of IBM WebSphere Application Server - Express V5.0.2, with additional support for ORB and EJBs []Support for SOAP 1.1, WSDL 1.1, UDDI 2.0, JAX-RPC 1.0, EJB 2.0, Enterprise Web Services 1.0 (JSR 109), WSDL4J, UDDI4J, and WS-Security []IBM WebSphere UDDI v2.0 registry []An entry-level database providing a JDBC implementation []IBM SDK for Java 2 Standard Edition (J2SE) Technology, version 1.3.1 []Eclipse plug-ins to expose JavaBeans and stateless session EJBs as Web services, to enable browsing for Web services in UDDI registries, to create Web services from WSDL definitions, and to publish and unpublish Web services to a UDDI registry []Command line tools to expose JavaBeans and stateless session EJBs as Web services, to create Web services from WSDL definitions, and to publish and unpublish Web services to a UDDI registry [*]Samples showing how to:
    [list=1]
  • expose JavaBeans and stateless session EJBs as Web services []create Web services from WSDL definitions []publish, unpublish and lookup Web services using UDDI [*]create secure Web services using the WS-Security specification
    [/list]
  • Information on how to use a Supply Chain Management Sample that is WS-I conformant [*]Comprehensive documentation including Web services concepts, developer tasks, and reference materials

See also the main IBM developer kits page.

Mark

37

Hi Mark,
Aloha. In an attempt to get myself off the DeadHorseSociety :wink: I ran some load test on webMethods IS 6.1 running IBM jvm 1.3.1 on Windows 2000 (my local box, couldn’t get any of our development boxes to set up this test)

I ran a simple load test (with and without GC scheduled) which simply invokes a web service; I also set up a separate scheduler which logs available memory by invoking WmRoot Svc getResourceSettings.

Generally I have GC scheduled to run on production instances every 5 min but for this load test, I reduced it to run every minute.

Case 1 : (With GC Scheduled @ 1 minute)

The Available memory starts with 86%, reduces to 80% to 73%, GC kicks in, the available memory bumps up to 86%… and this cycle continues.

2004-08-16 18:06:22 EDT I: 86 % (338610 KB)
2004-08-16 18:06:23 EDT I: 86 % (337663 KB)

2004-08-16 18:06:47 EDT I: 81 % (317329 KB)
2004-08-16 18:06:48 EDT I: 80 % (316286 KB)

2004-08-16 18:07:16 EDT I: 74 % (290402 KB)
2004-08-16 18:07:21 EDT I: 73 % (285571 KB)
2004-08-16 18:07:22 EDT I: 73 % (285295 KB)
2004-08-16 18:07:23 EDT I: 86 % (338837 KB)
2004-08-16 18:07:24 EDT I: 86 % (338717 KB)

Thus in this case, the memory never goes below 73%;

Case 2 : (Without GC Scheduled)

The Available memory starts with 83%, periodically/gradually reduces to 52% to 27% to 6% to 1%. When the available memory is @ 1%, GC automatically kicks in, and the available memory bumps up to 83%… and this cycle continues. Here are the logs for it. Notice, that available memory was at 1% for about 3 seconds.

2004-08-16 17:30:24 EDT I: 83 % (324618 KB)
2004-08-16 17:30:25 EDT I: 82 % (324590 KB)

2004-08-16 17:30:28 EDT I: 52 % (204892 KB)

2004-08-16 17:30:31 EDT I: 27 % (105191 KB)

2004-08-16 17:31:59 EDT I: 6 % (22876 KB)
2004-08-16 17:32:01 EDT I: 5 % (21199 KB)
2004-08-16 17:32:05 EDT I: 4 % (17689 KB)
2004-08-16 17:32:10 EDT I: 3 % (13038 KB)

2004-08-16 17:32:14 EDT I: 2 % (9471 KB)
2004-08-16 17:32:19 EDT I: 1 % (4885 KB)
2004-08-16 17:32:20 EDT I: 1 % (4051 KB)
2004-08-16 17:32:21 EDT I: 1 % (3224 KB)
2004-08-16 17:32:22 EDT I: 83 % (326800 KB)
2004-08-16 17:32:23 EDT I: 83 % (325678 KB)
2004-08-16 17:32:25 EDT I: 82 % (323926 KB)

This definitely proves that GC works. There is no need specifically to schedule GC. This vindicates your position on this issue (needless to say that we never had any doubt about your position on this or on IBM jvm programmers to implement GC correctly!!)

But I wonder if during that 3 second time window, the IS receives several http invokes to a particular svc, which say is memory intensive code. Will IS throw OutofMemory exceptions or not? Will it possibly be much slower than normal response from IS?

I have no idea. What are your thoughts on that? I will see if I can simulate these conditions somehow and run some load test to prove either ways?

Meanwhile, I continue to have GC scheduled on my IS instances making available memory on the server even out over a period of time. In the above instance, @ 86%.

One other point to note, that even if I don’t have the load test running (with GC disabled) and a scheduler to log available memory (making the server virtually idle), the memory keeps sinking to 1% eventually to rise to 83%.

Hope all this throws some light on the issue.

  • Saurabh.
38

And Mark,
possibly thinking of a solution to satisfy everyone in and off Dead Horse Society, I came up with this idea. (with keeping the above post in mind)

Schedule a java service; do Runtime.freeMemory()/Runtime.totalMemory(); if the value is below a particular threshold (say 0.2), do System.gc();

This way at least we would not be doing GC every 5 minutes or so, but instead when say the available memory is only 20%.

This is kind of middle approach, with benefits and limitations of both sides of argument.

  • Saurabh.
39

Saurabh,

Great work on your benchmarking! I’m reasonably sure that if additional memory allocation requests had been received by the JVM when there was only 1% available memory that an incremental GC would have been triggered.

Depending on your JVM vendor and version there are quite a few options to allow you to configure the garbage collection behavior. IBM 1.3.0 had almost no options to tune GC behavior while Sun’s 1.4.2 has quite a few. See Tuning Garbage Collection with the 1.4.2 Java Virtual Machine for more details.

You should be able to use the JVM’s options to set thresholds without scheduling anything in IS.

You are officially, no longer a member of the DHS! Congratulations!

BTW, I was reading some release notes about a recent fix to IS and it stated that explicit calls to System.GC were removed from some webMethods flat file handling code to improve performance especially on multi-processor servers.

Mark

40

BTW, the real goal is not to use the least amount of memory, but to achieve the desired combination of fastest throughput and highest efficiency related to how much time the CPU spends doing useful work vs. running garbage collection tasks.

You could schedule System.GC every 10 seconds to keep memory utilization low, but would probably kill performance in the process.

Anyone interested enough in performance to start messing around with explicit calls to System.GC() should first carefully read Sun’s GC tuning article.