Integration Server Spontaneously Restarts

, ,
1

All,

Has anyone had an experience where the Integration Server spontaneously restarts? This has happened several times to us and doesn’t appear to be associated with any specific activity on the server. Our current theory is that the jvm is crashing and restarting the server.

We’re running IS4.6 on Windows NT with jdk1.3.1_08.

Thanks…

J Silver

2

Was it a graceful shutdown, or does your server die abruptly?

It would be handy to inspect the server.bat script to see how it handles a JVM crash. I’m not sure about Windows, but maybe you could redirect the output from server.bat to a file and check it to see if you get java.lang.OutOfMemory errors or something of that nature when your server restarts.

3

The shutdown wasn’t graceful… when looking at the server.log, it shows normal activity and then it shows the normal entries that occur when the IS is started. There is no indication of a shutdown.

I was under the impression that the output of server.bat goes to the server.log. If that’s true, then no java.lang.OutOfMemory errors occurred. We’ve seen those in the past, but it does not appear to be an issue now.

If there’s somewhere else I can go to look for the server.bat output, I’m open to suggestions. I should mention that we run the IS as an NT service and not by directly executing the server.bat.

Meanwhile, I’ll take a look at server.bat to see if I can figure out what it does with a JVM crash.

Thanks!

4

Josh,

I would suggest checking the NT event log and Dr Watson’s log, if that is enabled.

It is “normal” for a JVM crash to not leave any information in the IS logs, as the process that would have put them there is dead.

Howerer, following a JVM crash, the IS should NOT be running!

I would be more interested in how and why it restarted. How long after the crash did the restart occur? Unless you have some auto-restart code in the batch file (server.bat)(or some job scheduling/monitoring appliication restarts it) , the IS should not restart itself as you describe.

If the server in invoking it’s internal restart function, I would expect to see log entries to that effect.

Foster

5

Hello all,
to add flawour to this discussion, we have problem of restarting SAP Business Connector 4.6 during startup of the server (OS Win2k Server).
SBC is restarting several times and than working normally. I can see in Task manager how java memory is going up several times (now it needs three or four times to start working).
We reinstalled jdk1.3.1_08 and recreate runserver.bat but the problem still exist.

Gordan

6

> I was under the impression that the output of server.bat goes
> to the server.log. If that’s true, then no java.lang.OutOfMemory
> errors occurred.

That seems right. However, if my memory is correct, there have been times I’ve seen java.lang.OutOfMemory errors appear only on the console I ran the server from - the errors did not show up in server.log.

7

Foster: I would be more interested in how and why it restarted. How long after the crash did the restart occur? Unless you have some auto-restart code in the batch file (server.bat)(or some job scheduling/monitoring appliication restarts it) , the IS should not restart itself as you describe.

Response: We’re also interested in why the server is restarting itself and not just dying. I was hoping to find something in server.bat that would cause that to happen (I haven’t had time to look yet). I couldn’t think of any other reason why this would happen unless Windows was killing the service and then restarting it automatically for some reason.

The largest length of time I’ve seen in the server.log between the last known activity and the first line of the startup was about 3 minutes.

Thanks all…

8

All,

I just found this bit of information from one of my colleagues as well… any insight?

“runserver.bat is exiting with an errorlevel of 42. When that happens there is a go to “if ERRORLEVEL 42 goto start”. This restarts the IS. What does ERRORLEVEL 42 mean?? It must mean something or server.bat would not have the “if ERRORLEVEL 42 goto start” line in it. It appears to me that the the jvm is crashing - perhaps it is just exiting in whatever manner that causes an ERRORLEVEL 42 ???”

Thanks again…

9

System.exit(42);
is done by IS when a shutdown-with-restart service is called. a.k.a. bounce.

The first thing I would do is change the Administrator password.

10

hi Josh,
I’ve seen IS automatically restart, when maximum number of IDataCursor are opened in custom Java service. So, check your custom java services and destroy any opened cursors.

Hope this hint helps.

11

The Administrator password was changed at installation (quite a while ago). Just to be sure, I verified this morning.

I will look into the IDataCursor issue as well…

12

Umashanker: "I’ve seen IS automatically restart, when maximum number of IDataCursor are opened in custom Java service. So, check your custom java services and destroy any opened cursors.

Hope this hint helps."

Was this only an issue in custom Java services or is there a potential for automatic restarts when a Flow service’s pipeline is not completely cleared after execution?

13

I don’t think or seen IS restart due to pipeline clear, it happened only for excessive opening of IDataCursor in bug custom java service(infinite IDataCursor opened).

14

Thanks all for the input… if anyone has any more, feel free to post. I’ll update the chain with our solution once we get the problem fixed…

15

This issue has been resolved. We believe the error occurred because the jvm was misreading signals sent by the operating system. The resolution (also recommended by webMethods product support) is to add the attribute “-Xrs” to the command in server.bat that starts the jvm.

This attribute tells Java to ignore some of the operating system signals that may otherwise cause it to restart.

Enjoy…

16

For the curious out there, here is the doc from Sun on the -Xrs option:

-Xrs  
Reduces usage of operating-system signals by the Java virtual machine 
(JVM). This option is available beginning with J2SE 1.3.1.  
In J2SE 1.3.0, the Shutdown Hooks facility was added to allow orderly  
shutdown of a Java application. The intent was to allow user cleanup 
code (such as closing database connections) to run at shutdown, even if 
the JVM terminates abruptly.  
 
The JVM watches for console control events to implement shutdown hooks 
for abnormal JVM termination. Specifically, the JVM registers a console 
control handler which begins shutdown-hook processing and returns TRUE 
for CTRL_C_EVENT, CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, and 
CTRL_SHUTDOWN_EVENT.  
 
The JVM uses a similar mechanism to implement the pre-1.2 feature of 
dumping thread stacks for debugging purposes. Sun's JVM uses 
CTRL_BREAK_EVENT to perform thread dumps. 
 
If the JVM is run as a service (for example, the servlet engine for a 
web server), it can receive CTRL_LOGOFF_EVENT but should not initiate 
shutdown since the operating system will not actually terminate the 
process. To avoid possible interference such as this, the -Xrs command- 
line option has been added beginning with J2SE 1.3.1. When the -Xrs  
option is used on Sun's JVM, the JVM does not install a console control  
handler, implying that it does not watch for or process CTRL_C_EVENT,  
CTRL_CLOSE_EVENT, CTRL_LOGOFF_EVENT, or CTRL_SHUTDOWN_EVENT.  
 
There are two consequences of specifying -Xrs:  
 
Ctrl-Break thread dumps are not available.  
User code is responsible for causing shutdown hooks to run, for example  
by calling System.exit() when the JVM is to be terminated.