Our server crashes regularly (once every couple of weeks) due to OutofMemoryException (OOME); We are trying to identify the root cause of this issue. We have identified that a particular service (custom code) that we have developed was running at the time of the latest OOME. At this point, we are not sure if the application custom code that we have created is causing this issue. I would like to mention here that the service is being called by a ‘file polling’. Also, our server processes 10000 transactions per day, so it is not very high volume.
A report that we extracted from the server indicated the memory usage around 600 MB (max set to 1 GB) when the server went down. Our JVM is set to 1 GB.
Has anyone encountered the same issue? What is the resolution? Is this due to “custom code” or do we have to configure some setting on the system side?
Please advise; I thank you in advance for your valuable help.
Can you show us the full error log before the crash ?
Which Integration Server version are you using?
Sometimes OOME may be caused by a small (or no) value for PermSize and MaxPermsize
“The permanent generation is used to hold reflective data of the VM itself such as class objects and method objects. These reflective objects are allocated directly into the permanent generation, and it is sized independently from the other generations”
PermSize is additional to the -Xmx value set by the user on the JVM options. But MaxPermSize allows for the JVM to be able to grow the PermSize to the amount specified. Initially when the VM is loaded, the MaxPermSize will still be the default value (32mb for -client and 64mb for -server) but will not actually take up that amount until it is needed.
I managed in the past to get rid with the Permsize OOME by increasing the value of Permsize
The other thing to investigate is the size of the files that are being processed and the way that you’re processing them. If you’re using flat file parsing to process the documents you may need to look at using iteration. Iteration breaks large files down into smaller chunks so that the entire file isn’t loaded into memory at one time.
If you get a large file once every couple of weeks (once a month?) then that could be the trigger for your OOME.
Thank you for your quick response. I forwarded your response to our System admin and got the following response; Can you help me on this and give us some direction?; Thank you in advance:
Which log are you wanting? I can provide it to you if you can tell me.
Currently we are using the default values for Permsize which is for the JRE.
Good input but if its an issue with the Permsize or not or any other configuration of the JVM, we need the verbose Garbage collection output. Without this tuning the JVM could work against us and would be based on assumptions on the root cause. I would be getting this logging in after CCB this week. As rightly pointed out, “Your problem may be elsewhere but check first the entire log before the crash to know the source of OOME.”
And Perm space is the space JVM uses to load classes and objects. Hence, we do need to be conscious about the code we put in and make it most optimal.
Lets get the GC logs in and see how the JVM is doing and catch some trends.
Thanks for the response; There are multiple individual files within the folder, each with a size of 1 K. The folder is polled every 3 hours and the service to process these files is called; so to answer your question the service is processing a file of 1K at a time; however at the time of polling there may be numerous files within the folder - to the extent of 300 to 500 files of 1 K possible within the folder.
In general, java out of memory exception happens in at less three scenario:
no more heap
no more permsize
not enough swap
For situation 1 & 2, you can turn on verbose:gc to determine whether they are the case(of course, you will need to know how to interpret the gc log and find out the current settings on the max heap and max permsize. For the heap, some time adjusting the ratio between young and old generation will fix the issue).
For swap, it is usually happen during the startup time or when java application do a fork/exec on external command. If the work you fork only require 5MB of memory but the java xms and xmx size are set equally(best practice on production) with a 1GB for example, it will need 1GB of swap when the fork happens.
I am getting an error as out of swap space on our server.
This is causing an intermidiate outage of our server.
please guide me on where I can do the changes specified by you.
(For swap, it is usually happen during the startup time or when java application do a fork/exec on external command. If the work you fork only require 5MB of memory but the java xms and xmx size are set equally(best practice on production) with a 1GB for example, it will need 1GB of swap when the fork happens. )