Has anyone seen the JVM core dump

[Guest]

The JVM is core dumping with the following as the current thread:

 1XMCURTHDINFO  Current Thread Details
NULL           ----------------------
3XMTHREADINFO      "Cron Daemon Pool1" (TID:0x101EB468, sys_thread_t:0x65AD3E0, state:R, native ID:0x13E0) prio=5
4XESTACKTRACE          at com.wm.util.coder.RPCBinCoder2._encode(RPCBinCoder2.java:643)
4XESTACKTRACE          at com.wm.util.coder.RPCBinCoder2._encode(RPCBinCoder2.java:467)
4XESTACKTRACE          at com.wm.util.coder.RPCBinCoder2.encode(RPCBinCoder2.java:300)
4XESTACKTRACE          at com.wm.util.queue.FSValuesHash.writeEntry(FSValuesHash.java:335)
4XESTACKTRACE          at com.wm.util.queue.FSValuesHash.add(FSValuesHash.java:153)
4XESTACKTRACE          at com.wm.util.queue.FSMTValuesHash.add(FSMTValuesHash.java:271)
4XESTACKTRACE          at com.wm.util.queue.FSMTValuesHash.put(FSMTValuesHash.java:369)
4XESTACKTRACE          at com.wm.app.repo.LocalRepositoryContext.put(LocalRepositoryContext.java:205)
4XESTACKTRACE          at com.wm.app.repo.RepositoryContext.put(RepositoryContext.java:136)
4XESTACKTRACE          at com.wm.app.b2b.util.RepoHelper.putInRepo(RepoHelper.java:201)
4XESTACKTRACE          at com.wm.app.b2b.server.cluster.SessionStore.addSession(SessionStore.java:169)
4XESTACKTRACE          at com.wm.app.b2b.server.cluster.ClusterManager.addSession(ClusterManager.java:619)
4XESTACKTRACE          at com.wm.app.b2b.server.StateManager.run(StateManager.java:235)
4XESTACKTRACE          at com.wm.util.pool.PooledThread.run(PooledThread.java:105)
4XESTACKTRACE          at java.lang.Thread.run(Thread.java:568)

Has anyone seen this? Or even better found the cause or solution?

Pass

[wMusers.Com1 (wMusers.Com)]

Pass,

Please provide additional details such as which product, which JVM vendor and version, how much memory is allocated, etc.

Mark

[Guest]

We are running Intergration Server with the default JVM (J2RE 1.4.2 IBM Windows 32
build cn142-20040926). With the follow memory parameters:

2CIUSERARG -Xms1024M 0x10000000
2CIUSERARG -Xmx1024M 0x60EC6C
2CIUSERARG -XX:MaxPermSize=256m 0x60ECC8

For the sake of full disclosure, the header of the core dump follows.

Thank you for any help you can provide.

 NULL           ------------------------------------------------------------------------
0SECTION       TITLE subcomponent dump routine
NULL           ===============================
1TISIGINFO     signal 11 received 
1TIDATETIME    Date:                 2006/05/08 at 08:18:34
1TIFILENAME    Javacore filename:    d:\Program Files\webMethods6\IntegrationServer \javacore.20060508.081834.4680.txt
NULL           ------------------------------------------------------------------------
0SECTION       XHPI subcomponent dump routine
NULL           ==============================
1XHEXCPCODE    Exception code: C0000005 Access Violation
1XHEXCPADDRESS Fault address:  05CB4726 01:000A3726
1XHEXCPMODULE  Fault module:   D:\Program Files\webMethods6\IntegrationServer \jvm\bin\jitc.dll
NULL           
1XHREGISTERS   Registers:
2XHREGISTER      EAX:0613AFAC
2XHREGISTER      EBX:06176898
2XHREGISTER      ECX:00000001
2XHREGISTER      EDX:000002D6
2XHREGISTER      ESI:000003A1
2XHREGISTER      EDI:0612E35C
2XHREGISTER      CS:EIP:001B:05CB4726
2XHREGISTER      SS:ESP:0023:06B4F334  EBP:00000002
2XHREGISTER      DS:0023  ES:0023  FS:003B  GS:0000
NULL           
1XHFLAGS       Flags:00010202
NULL           
NULL           ------------------------------------------------------------------------
0SECTION       CI subcomponent dump routine
NULL           ============================
1CIJAVAVERSION J2RE 1.4.2 IBM Windows 32 build cn142-20040926
1CIRUNNINGAS   Running as a standalone JVM
1CIJAVAHOMEDIR Java Home Dir:   d:\Program Files\webMethods6\IntegrationServer\jvm
1CIJAVADLLDIR  Java DLL Dir:    d:\Program Files\webMethods6\IntegrationServer\jvm\bin
1CISYSCP       Sys Classpath:   d:\Program Files\webMethods6\IntegrationServer\jvm \lib\core.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\graphics.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\security.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\server.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\xml.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\charsets.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmcertpathprovider.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjaasactivelm.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjaaslm.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjcefw.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjgssprovider.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjssefips.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmjsseprovider.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmorb.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmorbapi.jar;d:\Program Files\webMethods6\IntegrationServer\jvm\lib\ibmpkcs.jar
1CIUSERARGS    UserArgs:
2CIUSERARG               -Xms1024M 0x10000000
2CIUSERARG               -Xmx1024M 0x60EC6C
2CIUSERARG               -XX:MaxPermSize=256m 0x60ECC8
2CIUSERARG               -Xrs 0x7C889D94
2CIUSERARG               -DWM_HOME=d:\Program 0x409C00
2CIUSERARG               -Djava.class.path=;d:\Program Files\webMethods6\IntegrationServer\ jvm\lib\rt.jar; d:\Program Files\webMethods6\IntegrationServer\jvm\lib\i18n.jar;d:\Program Files\ webMethods6\IntegrationServer\lib\proxy.jar;  0xFFFF
2CIUSERARG               -Djava.library.path=packages\WmWin32\code\libs;C:\WINDOWS\system32; C:\WINDOWS;C:\WINDOWS\System32\Wbem;
d:\Program Files\webMethods6\IntegrationServer\ ..\common\bin;d:\Program Files\webMethods6\IntegrationServer\support\win32; 
d:\Program Files\webMethods6\IntegrationServer\jvm\bin\classic; 
d:\Program Files\webMethods6\IntegrationServer\lib; 0x1485D8
NULL           
1CIJVMMI       JVM Monitoring Interface (JVMMI)
NULL           ------------------------
2CIJVMMIOFF    No events are enabled.
NULL           
NULL           ------------------------------------------------------------------------
0SECTION       DC subcomponent dump routine
NULL           ============================
1DCHEADEREYE   Header eye catcher  DCST
1DCHEADERLEN   Header length       24
1DCHEADERVER   Header version      1
1DCHEADERMOD   Header modification 0
1DCINTERFACE   DC Interface at 0x1015DB5C with 15 entries
2DCINTERFACE      1 - dcCString2JavaString           0x1006E820
2DCINTERFACE      2 - dcInt642CString                0x1006E9E0
2DCINTERFACE      3 - dcJavaString2NewCString        0x1006EC10
2DCINTERFACE      4 - dcJavaString2CString           0x1006EB30
2DCINTERFACE      5 - dcJavaString2NewPlatformString 0x1006ED60
2DCINTERFACE      6 - dcJavaString2UTF               0x1006F100
2DCINTERFACE      7 - dcPlatformString2JavaString    0x1006EE80
2DCINTERFACE      8 - dcUnicode2UTF                  0x1006F030
2DCINTERFACE      9 - dcUnicode2UTFLength            0x1006EFA0
2DCINTERFACE     10 - dcUTF2JavaString               0x1006F4D0
2DCINTERFACE     11 - dcUTFClassName2JavaString      0x1006F7D0
2DCINTERFACE     12 - dcJavaString2ClassName         0x1006F1D0
2DCINTERFACE     13 - dcUTF2UnicodeNext              0x1006FA50
2DCINTERFACE     14 - dcVerifyUTF8                   0x1006FB40
2DCINTERFACE     15 - dcDumpRoutine                  0x100701D0
1DCARRAYINFO   Array info at 0x101129A8 with 16 entries
2DCARRAYINFO      1 - index  0 signature 0 name      N/A factor 0
2DCARRAYINFO      2 - index  0 signature 0 name      N/A factor 0
2DCARRAYINFO      3 - index  2 signature L name  class[] factor 4
2DCARRAYINFO      4 - index  0 signature 0 name      N/A factor 0
2DCARRAYINFO      5 - index  4 signature Z name   bool[] factor 1
2DCARRAYINFO      6 - index  5 signature C name   char[] factor 2
2DCARRAYINFO      7 - index  6 signature F name  float[] factor 4
2DCARRAYINFO      8 - index  7 signature D name double[] factor 8
2DCARRAYINFO      9 - index  8 signature B name   byte[] factor 1
2DCARRAYINFO     10 - index  9 signature S name  short[] factor 2
2DCARRAYINFO     11 - index 10 signature I name    int[] factor 4
2DCARRAYINFO     12 - index 11 signature J name   long[] factor 8
2DCARRAYINFO     13 - index  0 signature 0 name   uint[] factor 0
2DCARRAYINFO     14 - index  0 signature 0 name  uint1[] factor 0
2DCARRAYINFO     15 - index  0 signature 0 name  uint2[] factor 0
2DCARRAYINFO     16 - index  0 signature 0 name  uint3[] factor 0
NULL           ------------------------------------------------------------------------
0SECTION       DG subcomponent dump routine
NULL           ============================
1DGTRCENABLED  Trace enabled: Yes
2DGTRCTYPE       Trace: Internal
2DGTRCBUFFERS    Buffer specification: 8k
2DGTRCBUFALLOC   Buffers allocated: 0
2DGTRCBUFUSED    Buffers in use:    0
1DGJDUMPBUFF   Javadump buffer size (allocated): 2621440
NULL           ------------------------------------------------------------------------
0SECTION       ST subcomponent dump routine
NULL           ============================
1STGCMODES     Resettable GC: No
1STGCMODES     Concurrent GC: No
1STCURHBASE    Current Heap Base: 0x101801FC
1STCURHLIM     Current Heap Limit: 0x5017FBFC
1STMWHBASE     Middleware Heap Base: 0x101801FC
1STMWHLIM      Middleware Heap Limit: 0x5017FBFC
1STGCHELPERS   Number of GC Helper Threads: 3
1STJVMOPTS     -Xconcurrentlevel: 0 
1STJVMOPTS     -Xconcurrentbackground: 0 
1STGCCTR       GC Counter: 10 
1STAFCTR       AF Counter: 10 
1STHEAPFREE    Bytes of Heap Space Free: 10bb3620 
1STHEAPALLOC   Bytes of Heap Space Allocated: 3ffffa00 
1STSMBASE      SM Base: 0x0 
1STSMEND       SM End: 0x0 
1STPAMSTART    PAM Start: 0x0 
1STPAMEND      PAM End: 0x0 
1STCOMACTION   Compact Action: 0 
NULL           ------------------------------------------------------------------------
0SECTION       XE subcomponent dump routine
NULL           ============================
1XETHRESHOLD   MMI threshold for java methods is set to 2000
1XEJITINIT     JIT is initialized 
1XEJVMPIOFF    JVMPI is not activated 
1XEJNITHRESH   MMI threshold for JNI methods is set to 0
1XETRCHIS      Trace history length is set to 4
1XEJITDUMP     JIT dump routine is not yet implemented.
NULL           ------------------------------------------------------------------------
0SECTION       LK subcomponent dump routine
NULL           ============================
NULL           
1LKPOOLINFO    Monitor pool info:
2LKPOOLINIT      Initial monitor count: 32
2LKPOOLEXPNUM    Minimum number of free monitors before expansion: 5
2LKPOOLEXPBY     Pool will next be expanded by: 409
2LKPOOLTOTAL     Current total number of monitors: 819
2LKPOOLFREE      Current number of free monitors: 143
NULL

[wMusers.Com1 (wMusers.Com)]

I ran across this myself recently and logged an SR with WM Support. I had already read through the core dumps to determine that IS seemed to be having an Just in Time Compiler (JITC) issue with a particular method. I followed the approach outlined in the attached PDF to identify the method that the JIT compiler was not able to process.

In my case the two core dump files showed the following:

• com.wm.util.coder.RPCBinCoder2._encode(RPCBinCoder2.java:467) – Invoked by “HTTP Handler ”
• com.wm.util.coder.RPCBinCoder2._encode(RPCBinCoder2.java:461) – Invoked by “Cron Daemon Pool2”

Based on the attached PDF, IBM recommends excluding the method from JITC using the following option:

JITC_COMPILEOPT=SKIP{com/wm/util/coder/RPCBinCoder2}{_encode}
 

Turns out that this is a known issue with IS 6.5 running as an NT Service
and is mentioned in the ReleaseNotes.html file that ships with IS 6.5.

@webMethods Release Notes

1-W19SX
If you are running the Integration Server with IBM JVM 1.4.2 on Windows as a Windows Service, you must set the following environment variable: JITC_COMPILEOPT=SKIP{com/wm/util/coder/RPCBinCoder2}{_encode}{java/lang/Thread}{sleep}
This environment variable is needed to work around a bug with the Just In Time compiler of the
IBM JVM 1.4.2 on Windows. If you start the Integration Server from the command line, you do not need to define this environment variable manually because it is already set in bin\server.bat.

See also this IBM Developerworks article for details on the JIT Compiler issue.

I added a new System-level environment variable named “” and set its value to the following:

SKIP{com/wm/util/coder/RPCBinCoder2}{_encode}{java/lang/Thread}{sleep}

Hopefully, this will do the trick.

Mark
Analyzing IBM JVM Core Dumps.pdf (396 KB)

[wMusers.Com1 (wMusers.Com)]

Related Critical Software Update

[steve90]

Hello guys,

As a follow-up to this question, I would like to clarify how I can generate a JVM core dump or hs_err_pid file.

I am running IS using a Sun JVM but it does not generate such files.

The IS runs as a Service on a Windows 2000 server and the only option seems to be as follows:-

1. stop IS

2. remove -Xrs switch from server.bat file

3. start IS in foreground on Windows server

4. wait for crash

However, I would like to be able to perform this with less manual input and maybe set a switch on the IS start command to produce a JVM dump file without staying logged on to the server!!!

Thanks in advance,

Steve

[wMusers.Com1 (wMusers.Com)]

You can generate a thread dump (not technically a core dump since contents of memory are not written to a file) by sending a “kill -3 process_id” to a unix process or by pressing CTRL-\ in Unix or CTRL-BREAK from the Windows command prompt.

See this thread for more.

Mark

[Yemi_Bedu]

Hello,
I am just getting the time to look at this and have applied the env variable fix to see if that helps while considering upgrading to a new jvm. We are running included ibm 1.4.2 (48.0) with is 6.5 sp3 build 394.

Will this be an option to take out when switching over to later version or different vendor jvm? Thank you. Good day.

Yemi Bedu