hello,all.
We are using IS 6.1 on HP-UNIX. And the Max Java Memory size is 1.5G. Everything works well right after the IS rebooting. But after a period, say 1 month, the memory usage can be 90 percent and the server reacts really slowly. When we came into this situation, the only thing we can do is reboot. But i think that is not the right way.
It seems there is something wrong with memory recollection and maybe we should do some changes on our flow service.
Is there something more we can do to solve this problem?
Any comments are highly appreciated.
BTW There are no error message on server log
Thanks.
Your statement that “the main cause of your problem is the OS” is at best premature and quite possibly very misleading.
Nancy’s question was not about how she could add more memory to the JVM heap size. Also, most sensible customers won’t throw out a perfectly good, very expensive server just to increase memory for one application.
Nancy, high memory usage itself is not a problem and may or may not be related to the perceived slow response in the server. You would probably benefit by finding someone who understands java garbage collection (GC) tuning and having them assist you with using the verboseGC JVM option with HP’s JTune to understand what is going on inside the JVM when these slowdown’s occur.
I would also have someone familiar with HP’s software patch process help you track down the latest patces for the HP JVM that you are using. There have been issues in the past with HP’s java that were addressed by applying the correct patches. Don’t start there, but be aware of this situation.
To temporarily resolve the problem we scheduled a service that runs java System.gc(); I cannot say that this solves the problem. But in general, to my experience, it will reduce your “restart IS” frequency.
We also applied the latest patch of the HP-1.4.2 JVM and increase the Min heap to make it the closer with the max heap. (We use 1024M). This will in general increase the frequency of the Garbage Collection. You have to be cautious, because running gc to often may reduce your performance. This is where tuning will probably takes place. So, it is largely depends on your application and how heavy is your IS usage, in terms of mapping etc.
My best suggestion is to go through your services again. Most of the time you will probably find some unused pipeline that didn’t get cleared, db connection from wmDB don’t get closed properly, disabled flow-steps etc.
I feel that optimizing your services may reduce your memory problems or even solve it.
IS slow response sometimes also relates with the frequency of pub-sub of IS with the brokers, your message delivery mechanism and the size of the docs that get pass around. If you use workflow with large docsuments and comlicated hierarchical mapping, the IS of the workflow could potentially be running real slow, especially if you use WMTomcat to deploy your workflow.
So, my suggestion is look at your architecture of how do you use the IS and check all the services to make sure everything clean and in prod, remove all the disabled steps as well.
Thanks Mark and Kurt.
I have a question about the pipeline data. Will all the pipeline data be cleared when the service ends? If so, what will be the most likely data taken the memory?
Thinking about the memory leaking, will this be a proper reason for this situation. Because it is said jvm 1.4 has already solved this problem. If so, what will be the possible reason.
And Mark, thank you for your good advice. But right now, we don’t have such java garbage collection (GC) tuning experts. And i’ll focus on these part recently. So could you give more detail in this area? maybe books or guidelines or some tools to help do the analyze. And by the way, we have the license for Load Runner to do the stress test. But i’m not sure whether this will help to find the problem. Any advice?
BTW, we don’t use broker. We are using wm trigger to accept the document sent from GXS enterprise
Thankyou
Nancy
Kurt, you’ve mentioned this:wmDB don’t get closed . In our service, for each db operation, we use starttransaction and commit transaction, will this close the db connection? or is there any situation that leave the db connection open?
Thanks!
Nancy
It’s funny that you asked this question because just recently I began investigating this exact issue. In my past experience with the webMethods IS, I’ve never had to use WmDB. All databases I had dealt with were supported by the JDBC adapter. However, just recently I got involved in a project for which the database is not supported by the JDBC adapter so I’m seeing myself having to get familiar with WmDB.
So far, based on tests that I have been running, if you have WmDB set to do connection pooling (see the WmDB guide for details), you MUST close (pub.db:close) each connection after you’re done. If you do NOT, that connection will not be made available again until it expires. This means that IF you’re opening several connections and not closing them, THEN once you reach the maximum number of connections allowed for that Alias, every service that attemps to use a connection will block and will have to wait until one of the “unclosed” connections expires. As you can guess, this will impact performance tremendously.
Again, I am a WmDB newbie and these were observations that I have just recently made. If any of you out there know that I have misstated something, please let me know.
We’ve discussed at length here, although not for some time, the fact that the JVM vendors recommend that you do not schedule System.GC() as it works against the very sophisticated garbage collection algorithms.
Nancy, you can enable verboseGC logging by simply adding a parameter to the command that starts java in your server.sh or server.bat file. The command will write to an output file. Tools like HP’s JTune will read this file and graph the major and minor garbage collection events as well as the amount of memory in use and reclaimed in each GC. This can show you how much memory is really being used and how much of your processing capacity is being used to perform GC activities.
Capturing verboseGC information is relatively simple to setup. Producing graphs of GC activity is also simple using a variety of tools. To quote the credit card commercial “Finding someone who can interpret the graphs… priceless.”
Percio, thank you for ur explanation om WmDB. I think i just misunderstood. We are using JDBC for all the services. And if so, is there any problems about the unclosed connections?
Nancy
Mark, thank you.
I just found out that we havn’t used the watt.server.cache.gcMins in extend parameters. Do you think we’d better set this? or if not, does that mean there is no gc running on the IS server? But i am a little confused about this gc function. The JVM should do the GC job, right? So how does this watt.server.cache.gcMins do in IS? Is there any relations between them?
watt.server.cache.gcMins is not related to JVM gc. The ‘garbage collection’ referred to here is a cleanup of expired cache entries that are no longer valid as cache entries, but which may still be referenced in the Cache Manager storage.
Kurt, could you please talk about the last patch of Hp unix JVM1.4.2? Since we are using the same JVM, i’d like to know more about the patch.
What’s this patch about? Solved what problem? I just couldn’t find out the related info.
BTW, we don’t use service cache right now. So can i say this watt.server.cache.gcMins parameter is useless for us?
And for cache function, there is a cache expire setting.That’s the difference between the expire time and watt.server.cache.gcMins time?
Thank you!
nancy
If you use the JDBCAdapter (note: WmDB also uses JDBC to talk to the database), then you don’t have to explicitly open or close any connections, so you shouldn’t have any problems there.
However, although I have not personally experienced this, I have read/heard of people who have come across problems with the JDBC adapter when handling explicit transactions. If you’re suspicious that you’re db transactions are related to your performance problems, I suggest searching WMUsers for threads that talk about doing explicit transactions through the JDBC adapter.
However, you did not mention anything about invoking rollbackTransaction if an error occurs. Do you know if you’re code is handling exceptions appropriately? If not, then this could have a serious impact on your system.
From the IS Administrator’s Guide, watt.server.cache.gcMins ‘Specifies how often (in minutes) the server sweeps the cache to perform garbage collection’. The expire time on the service is how the server determines that a service cache has expired.
For additional information search your Admin Guide and Advantage for watt.server.cache.gcMins.
I was having memory problems similar to yours and took the steps recommended by Mark (turning on verbose GC and using HP Jtune to graphically analyze the results). Although my problems turned out to be unrelated to the JVM, this proved to be an extremely valuable exercise. I use Sun’s JVM but I found the following link on HP’s site that includes the startup setting to turn on verbose GC logging and a link to download HP Jtune: [url]http://www.hp.com/products1/unix/java/infolibrary/prog_guide/diag_memory_leaks.html#diagnosing[/url]
Even if you don’t have any “gurus” on site to analyze the output, you can often easily see exactly when memory usage spikes occur and look into what the IS was doing at those times. There have also been several threads in wMUsers on these topics that you should be able to find with the search function. HTH,