Applinx Tweaking in Linux

I would appreciate any ideas with the following:

We run Applinx 5.2.4 in a Linux partition on an IBM mainframe. Also Suse 9, Apache Tomcat 5.0.19, and IBM Java 1.4.2. Some of our Applinx enabled programs return a huge amount of data for a table to be displayed. Every so often we get a Java OutOfMemory error.

In the catalina.sh file we have:
CATALINA_OPTS=" -server -Xms2048m -Xmx2048m -Xincgc -Xcompactgc"

In the start-gxserver.sh file we have:
$JAVA_HOME/bin/java -Xmx1024m -Xcompactgc -Xbootclasspath/p:"$_BCP"
-classpath “$_CP” “-Dcom.sabratec.gxhome=$GXHOME”
“-Dcom.sabratec.license=$SAG/common/LKey/”
com.sabratec.applinx.server.runtime.GXServerContextStart &

Total RAM for the Linux partition is 4024mb.

What would be the most efficient way to maximize my heap size to give it the most space to allow for our huge data tables? I have tried many variations, but the above seems to be the best. Since I am no Linux expert, I know it can be improved. Suggestions are most welcome.

Thanks,

Joe

Hi Joe,
I remember this case.
Last time you told me you cannot upgrade the JVM because of OS limitation. Is that still the case?
According to this article - http://www-03.ibm.com/servers/eserver/zseries/software/java/

IBM has JVM 1.6 and 1.5 for z/os. If you can please upgrade your JVM

Hi Asaf,

I am trying to get our Mainframe outsourcer to upgrade the Java, but I am having some resistance. I will get it done, just not immediately. I have zipped the logs and the gxz as you requested, but I am getting this message:
“Sorry, you have reached your maximum Upload Quota Limit of 2 MB”. The file is only 258KB. How can I get the zipped fille to you?

Thanks,

Joe

Never mind. I deleted some old items and got the attachment up.
Logs.zip (258 KB)

Hi Joe,
I cannot see the zipped file - were you able to upload it?

Now I can.
I’ve download the files and we will take a look and let you know.
Thanks.

Hi,
We saw here that Tomcat got out of memory.
Please send us the files

Hi Asaf,

Unfortunately, the dumps are reviewed and then deleted. The space they take up on the Linux VM is too much to maintain. The heap dumps are very large. I can grab the next batch and make sure they are not deleted. I will also try to recreate it on the test VM. It should not be a problem. I will let you know the size and we can then decide how best to view it.

Thanks,

Joe

Ok, I have recreated the dumps, but the zipped file size is 41,741KB. Can you provide me with a Software AG server to load it to? Let me know what you decide. I have attached the new logs and the javacore dump. The appdump.gif file is the result in the browser, and the heaping.gif file shows the Linux status at the time of the dump.

Joe
Corelogs.zip (257 KB)

Asaf,

Did you get this reply?

Hi,
I only got the replay now.
I don

Hi Asaf,

I used Rapidshare for the large dump. The link is:

http://rapidshare.com/files/182776447/dumps.zip.html

The smaller files and logs were in the Corelogs.zip file attached to the previous reply of Wed. Nov 26, 2008.

Thanks,

Joe

Hi,
After investigating your files there are 2 things I want you to do:

  1. It seems that your web application is missing important Jar (castor-0.9.6b.jar) in path
    castor-0.9.6b.jar (1.22 MB)

Hi Asaf,

Well, it seems that I already have the castor-0.9.6b.jar file in the place where you indicated. Also, I did have the -Xmx parameter in the catalina.sh file, which I have attached. I will get you the core file when I get the next dump. I will crash the test system today and get you the files.

Thanks,

Joe
catalina.zip (2.56 KB)

Hi,
Maybe the Jar is corrupted? Try switching it with the one I sent you.
Maybe it’s better if you zip together all the relevant files (after you get the system to crush) and send it to me again. (using Rapidshare?)
Thanks,
Asaf

Asaf,

All of the dumps and logs can be retrieved from Rapid Share. There are 3 files and the addresses are:

http://rapidshare.com/files/183751658/logs_Jan15.zip

http://rapidshare.com/files/183751660/javacore_Jan15.zip

http://rapidshare.com/files/183751663/heapdump_Jan15.zip

Thanks,

Joe

Got the files, thanks a lot.
Our expert for this subject will be here only later this week but I will try to give you an answer as fast as I can.
Asaf

Hi again,
We want to do two things.
One would be fix the problem with the Jar file (

Here are the results of the commands you suggested:

dataprop:/SAG/sagprod/apx/v525/log # ps auxwww | grep catalina
tomcat 1708 0.2 15.0 778220 618636 ? S 03:06 0:44 /usr/lib/IBMJava2
-1.4.2/bin/java -Djava.endorsed.dirs=/usr/share/tomcat/common/endorsed -classpat
h /usr/lib/IBMJava2-1.4.2/lib/tools.jar:/usr/share/tomcat/bin/bootstrap.jar:/usr
/share/tomcat/bin/commons-logging-api.jar -Dcatalina.base=/srv/www/tomcat/base -
Dcatalina.home=/usr/share/tomcat -Djava.io.tmpdir=/srv/www/tomcat/base/temp org.
apache.catalina.startup.Bootstrap start
root 1883 0.1 6.9 1324432 285036 ? S 03:06 0:42 /usr/lib/IBMJava2
-1.4.2/bin/java -Xmx1024m -Xcompactgc -Xbootclasspath/p:/SAG/sagprod/apx/v525/li
b/xalan-2.6.0.jar:lib/xercesImpl-2.6.2.jar:lib/xml-apis-1.2.01.jar -classpath /S
AG/sagprod/apx/v525/lib/gxserver.jar:lib/activation-1.1.jar:lib/appserver/catali
na-optional.jar:lib/appserver/catalina.jar:lib/appserver/commons-el.jar:lib/apps
erver/commons-logging.jar:lib/appserver/commons-modeler.jar:lib/appserver/jasper
-compiler-jdt.jar:lib/appserver/jasper-compiler.jar:lib/appserver/jasper-runtime
.jar:lib/appserver/jmx.jar:lib/appserver/jsp-api.jar:lib/appserver/naming-factor
y.jar:lib/appserver/naming-resources.jar:lib/appserver/servlet-api.jar:lib/appse
rver/servlets-default.jar:lib/appserver/tomcat-coyote.jar:lib/appserver/tomcat-h
ttp.jar:lib/appserver/tomcat-util.jar:lib/axis2/XmlSchema-1.2.jar:lib/axis2/anno
gen-0.1.0.jar:lib/axis2/axiom-api-1.2.jar:lib/axis2/axiom-dom-1.2.jar:lib/axis2/
axiom-impl-1.2.jar:lib/axis2/axis2-adb-1.1.jar:lib/axis2/axis2-adb-codegen-1.1.j
ar:lib/axis2/axis2-kernel-1.1.jar:lib/axis2/commons-codec-1.3.jar:lib/axis2/comm
ons-httpclient-3.0.1.jar:lib/axis2/jakarta-httpcore-4.0-alpha2.jar:lib/axis2/nee
thi-2.0.jar:lib/axis2/stax-api-1.0.1.jar:lib/axis2/wstx-asl-3.0.1.jar:lib/c3p0-0
.9.0.jar:lib/commons-discovery-0.2.jar:lib/commons-logging-1.0.4.jar:lib/derby-1
0.1.jar:lib/j2ssh-core-0.2.2.jar:lib/jakarta-regexp-1.3.jar:lib/jregex1.2_01.jar
:lib/jt400.jar:lib/log4j-1.2.9b.jar:lib/mail-1.4.jar:lib/saaj.jar:lib/saglic.jar
:lib/servletapi-2.3.jar:lib/tools.jar:lib/wsdl4j-1.6.2.jar:lib/xstream-1.1.2.jar
-Dcom.sabratec.gxhome=/SAG/sagprod/apx/v525 -Dcom.sabratec.license=/SAG/sagprod
/common/LKey/ com.sabratec.applinx.server.runtime.GXServerContextStart
root 2836 0.0 0.0 1888 684 pts/3 R+ 09:08 0:00 grep catalina
dataprop:/SAG/sagprod/apx/v525/log #

dataprop:/SAG/sagprod/apx/v525/log # ps -fade | grep catalina
tomcat 1708 1 0 03:06 ? 00:00:44 /usr/lib/IBMJava2-1.4.2/bin/java
-Djava.endorsed.dirs=/usr/share/tomcat/common/endorsed -classpath /usr/lib/IBMJ
ava2-1.4.2/lib/tools.jar:/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/b
in/commons-logging-api.jar -Dcatalina.base=/srv/www/tomcat/base -Dcatalina.home=
/usr/share/tomcat -Djava.io.tmpdir=/srv/www/tomcat/base/temp org.apache.catalina
.startup.Bootstrap start
root 1883 1 0 03:06 ? 00:00:42 /usr/lib/IBMJava2-1.4.2/bin/java
-Xmx1024m -Xcompactgc -Xbootclasspath/p:/SAG/sagprod/apx/v525/lib/xalan-2.6.0.j
ar:lib/xercesImpl-2.6.2.jar:lib/xml-apis-1.2.01.jar -classpath /SAG/sagprod/apx/
v525/lib/gxserver.jar:lib/activation-1.1.jar:lib/appserver/catalina-optional.jar
:lib/appserver/catalina.jar:lib/appserver/commons-el.jar:lib/appserver/commons-l
ogging.jar:lib/appserver/commons-modeler.jar:lib/appserver/jasper-compiler-jdt.j
ar:lib/appserver/jasper-compiler.jar:lib/appserver/jasper-runtime.jar:lib/appser
ver/jmx.jar:lib/appserver/jsp-api.jar:lib/appserver/naming-factory.jar:lib/appse
rver/naming-resources.jar:lib/appserver/servlet-api.jar:lib/appserver/servlets-d
efault.jar:lib/appserver/tomcat-coyote.jar:lib/appserver/tomcat-http.jar:lib/app
server/tomcat-util.jar:lib/axis2/XmlSchema-1.2.jar:lib/axis2/annogen-0.1.0.jar:l
ib/axis2/axiom-api-1.2.jar:lib/axis2/axiom-dom-1.2.jar:lib/axis2/axiom-impl-1.2.
jar:lib/axis2/axis2-adb-1.1.jar:lib/axis2/axis2-adb-codegen-1.1.jar:lib/axis2/ax
is2-kernel-1.1.jar:lib/axis2/commons-codec-1.3.jar:lib/axis2/commons-httpclient-
3.0.1.jar:lib/axis2/jakarta-httpcore-4.0-alpha2.jar:lib/axis2/neethi-2.0.jar:lib
/axis2/stax-api-1.0.1.jar:lib/axis2/wstx-asl-3.0.1.jar:lib/c3p0-0.9.0.jar:lib/co
mmons-discovery-0.2.jar:lib/commons-logging-1.0.4.jar:lib/derby-10.1.jar:lib/j2s
sh-core-0.2.2.jar:lib/jakarta-regexp-1.3.jar:lib/jregex1.2_01.jar:lib/jt400.jar:
lib/log4j-1.2.9b.jar:lib/mail-1.4.jar:lib/saaj.jar:lib/saglic.jar:lib/servletapi
-2.3.jar:lib/tools.jar:lib/wsdl4j-1.6.2.jar:lib/xstream-1.1.2.jar -Dcom.sabratec
.gxhome=/SAG/sagprod/apx/v525 -Dcom.sabratec.license=/SAG/sagprod/common/LKey/ c
om.sabratec.applinx.server.runtime.GXServerContextStart
root 2838 2824 0 09:08 pts/3 00:00:00 grep catalina
dataprop:/SAG/sagprod/apx/v525/log #

I will work on the rest now.

Thanks,

Joe

On the jar file, it was never missing. It was located where you had indicated previously. I did replace it with the one you sent and there was no change in operations. I created a new application via the framework manager, and it also worked fine. My assumption falls back to our current version of Java, which is IBM 1.4.2. I believe it has memory leaks and probably other problems that would be fixed if we could upgrade it to the most current version. Unfortunately, since we use an outsourcer for our mainframe, it has been a struggle to get them to even look at upgrading it. I am pushing forward with it, but it will take time. I was just hoping for some possible tweaks to improve our chances of avoiding the “Out of Memory” problem. I do appreciate your assistance in trying to help me improve our Linux processing. Let me know if you would like any further info.

Thanks,

Joe

I also think that upgrading the Java version could solve the problem.
For now we can clearly see that the Tomcat is called without the -Xmx parameter according to what you

Asaf,

I have attached a zip file with the files I collected for you. It all seems fine to me. If I assign a max size of -Xmx2048m to Tomcat (in catalina.sh) , and then a max size of -Xmx1024m to the gxserver (in start-gxserver.sh), I think it should function correctly, but I’m just not sure. No changes have been made to anything in quite awhile (just me tweaking the catalina.sh and start-gxserver.sh files).

Thanks,

Joe
Files.zip (10.5 KB)