I been troubleshooting some out of memory errors in IS 6.5 SP2. These aren’t your normal running out of heap space type of memory issues. These out of memory errors are occurring in the non-heap space specifically in the PermGen memory space. JVM is Sun’s 1.5.0_10.
The PermGen space is reserved for class metadata and interned strings. Since I have never seen an java.lang.OutofMemory: PermGen space failure before, I had to do a little research on what the heck it was and why was it running out of memory.
I still don’t have the answer to why it is running out of memory but I’ll share what I know so far. If you have run into this and have some advice please jump right in.
The PermGen space is allocated 64M by default in Sun’s 1.5.0_10. It can be adjusted up by using these switches: -XX:PermSize and -XX:MaxPermSize.
There is some debate about whether garbage collection occurs in this space, but I have successfully tested that it does. Using a tool that is bundled in 1.5 called JConsole, I monitored the affected of load and unloading packages in the IS server.
For the test I simply reloaded the WmMonitor package numerous times until the PermGen memory limit was reached. At that time the Garbage collector kicked in and reduced the memory footprint back down to normal.
I found some discussions on the following switches:
-XX:+UseConcMarkSweepGC
-XX:+CMSPermGenSweepingEnabled
-XX:+CMSClassUnloadingEnabled
I ran the test with these enabled. I found no difference as to when the Garbage collector kicked in.
I suspect some sort of dangling reference is causing some classes not to get unloaded correctly when they should. I going to pursue some thread dumps to see if I can find any danglers. If anyone has run into this before, please share your insights.
Mark,
Hoping you can help me…
Given that you have some experience in monitoring IS using Jconsole,
I was wondering where in the server script you load the JMX agent (and how).
I am trying to get this working in both 7.1.1 and 6.5 SP2.
I need to perform the same as above, and use Jconsole on a Win2k3 box with IS running as a service. And when I type in jps, the running service doesn’t show a PID.
This is my config in server.bat. I am running IS 7.1.
set JAVA_MEMSET=-Xms%JAVA_MIN_MEM% -Xmx%JAVA_MAX_MEM% -XX:MaxPermSize=%JAVA_MAX_PERM_SIZE%
set JAVA2_MEMSET=-Xms%JAVA_MIN_MEM% -Xmx%JAVA_MAX_MEM% -XX:MaxPermSize=%JAVA_MAX_PERM_SIZE% -Xrs %XPATH_IMPL% -DWM_HOME=“%WM_HOME%” -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=7999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
I am also facing the “PermGen Space” error in our production environment. We have our webMethods server setup in windows operating system. The RAM size is around 12GB.
Server processes around 800+ transactions per day, few of them are having size of 200+ MB also.
We have scheduled our server to restart once in a week, but whenever the restart service runs server gets “hanged up” and “PermGen Space” is thrown.
Could anyone please let advice me, what could be the possible cause for it.
Kapil,
Are you using the 64bit edition? Just curious because of the amount of memory on your server which is not addressable by 32bit systems.
200MB is pretty big for a single transaction, although in theory that shouldn’t cause your permgen errors. You are probably going to have to provide us with a lot more details on your system in order for us to make an educated guess. webMethods IS version, configuration, JVM version, size of JVM, permgen size. How these 200MB transactions are handled?
And was that 800 transactions a day or 800k transactions a day?
Kapil,
Your config doesn’t look strange assuming you are using the 64bit version of the JVM and webMethods IS. You are only having this permgen issue on shutdown? What happens if you don’t shutdown every week? Are you seeing memory leaks?
