IS automatically gets restarted

, ,
1

IS has restatred automatically throwing no exceptions and errors at the time of restart. I am pasting the server log till the time it has got restated.
Somebody please surrest the reasons as this is happening again and again in production server.

2007-08-01 00:05:20 EST [ISP.0090.0003C] no xml found
2007-08-01 00:05:28 EST [ISP.0090.0003C] *** *** *** Initialize new server log *** *** ***
2007-08-01 00:06:45 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
2007-08-01 00:10:30 EST [ISP.0090.0004C] [NCS HouseKeeping] – Executed : /bin/ksh /bin/ksh /opt/webMethods6/ASPAC/CNS/IntegrationServer/packages/wM_JNJ_Utils/script/compress_cleanup.ksh “/opt/webMethods6/ASPAC/CNS/IntegrationServer”
2007-08-01 00:11:25 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
…
2007-08-01 00:29:00 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
2007-08-01 00:30:02 EST [ISP.0090.0003C] no xml found
2007-08-01 00:31:36 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
2007-08-01 00:35:13 EST [ISP.0090.0003C] no xml found
2007-08-01 00:38:20 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
…
2007-08-01 00:44:08 EST [ISP.0090.0003C] portalUser = dali-donghua-phj
2007-08-01 01:30:13 EST [ISP.0090.0003C] no xml found
2007-08-01 01:41:22 EST [ISU.0000.9999W] An exception occurred invoking a processing service for document 5d24630v2gef0g8e00001fpg. The processing service was CA105_XML_v1.services.Lotus:handleOrder, and the original error message was: com.wm.lang.flow.FlowException: Order validation failed. [TRNSERV.000014.000027]
2007-08-01 01:41:22 EST [ISU.0000.9999W] com.wm.lang.flow.FlowException: Order validation failed
2007-08-01 01:41:25 EST [ISU.0000.9999W] An exception occurred invoking a processing service for document 5d24630v2gef0jo100001fpj. The processing service was CA105_XML_v1.services.Lotus:handleOrder, and the original error message was: com.wm.lang.flow.FlowException: Order validation failed. [TRNSERV.000014.000027]
2007-08-01 01:41:25 EST [ISU.0000.9999W] com.wm.lang.flow.FlowException: Order validation failed
2007-08-01 01:41:28 EST [ISU.0000.9999W] An exception occurred invoking a processing service for document 5d24630v2gef0mgk00001fpm. The processing service was CA105_XML_v1.services.Lotus:handleOrder, and the original error message was: com.wm.lang.flow.FlowException: Order validation failed. [TRNSERV.000014.000027]
2007-08-01 01:41:28 EST [ISU.0000.9999W] com.wm.lang.flow.FlowException: Order validation failed
2007-08-01 01:50:13 EST [ISP.0090.0003C] no xml found
2007-08-01 02:13:09 EST [ISP.0090.0004C] [wM_JnJ_Utils] – Archiving “processArchive Done” : Processes have been successfully deleted from ProcessAudit runtime tables., Error : %errorMessage%
2007-08-01 03:18:56 EST [ISP.0090.0003C] orderDocFlow - USER = 119787 portalUser = tongren-boyu
2007-08-01 03:18:56 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:18:56 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:18:59 EST [ISP.0090.0003C] DELIVERY STATUS 0032609755
2007-08-01 03:18:59 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:00 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:00 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:01 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:02 EST [ISP.0090.0003C] DELIVERY STATUS 0032609757
2007-08-01 03:19:02 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:02 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:02 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:03 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:04 EST [ISP.0090.0003C] DELIVERY STATUS 0032642972
2007-08-01 03:19:04 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:04 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:05 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:06 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:06 EST [ISP.0090.0003C] DELIVERY STATUS 0032669787
2007-08-01 03:19:06 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:06 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:06 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:08 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:08 EST [ISP.0090.0003C] DELIVERY STATUS 0032669789
2007-08-01 03:19:08 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:08 EST [ISP.0090.0003C] orderDocFlow - USER = 119787 portalUser = tongren-boyu
2007-08-01 03:19:08 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:09 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:09 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:10 EST [ISP.0090.0003C] DELIVERY STATUS 0032609755
2007-08-01 03:19:10 EST [ISP.0090.0003C] portalUser = tongren-boyu
…
2007-08-01 03:19:11 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:11 EST [ISP.0090.0003C] DELIVERY STATUS 0032680289
2007-08-01 03:19:11 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:12 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:12 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:12 EST [ISP.0090.0003C] DELIVERY STATUS 0032609757
2007-08-01 03:19:12 EST [ISP.0090.0003C] portalUser = tongren-boyu
…
2007-08-01 03:19:13 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:14 EST [ISP.0090.0003C] DELIVERY STATUS 0032702331
2007-08-01 03:19:14 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:14 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:14 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:15 EST [ISP.0090.0003C] DELIVERY STATUS 0032642972
2007-08-01 03:19:15 EST [ISP.0090.0003C] portalUser = tongren-boyu
…
2007-08-01 03:19:17 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:17 EST [ISP.0090.0003C] DELIVERY STATUS 0032669787
2007-08-01 03:19:17 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:17 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:18 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:18 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:19 EST [ISP.0090.0003C] DELIVERY STATUS 0032669789
2007-08-01 03:19:19 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:20 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:21 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:21 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:22 EST [ISP.0090.0003C] DELIVERY STATUS 0032680289
2007-08-01 03:19:22 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:22 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:22 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:23 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 03:19:23 EST [ISP.0090.0003C] DELIVERY STATUS 0032702331
2007-08-01 03:19:23 EST [ISP.0090.0003C] portalUser = tongren-boyu
…
2007-08-01 03:56:36 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:11:00 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:12:14 EST [ISP.0090.0003C] orderDocFlow - USER = 119787 portalUser = tongren-boyu
2007-08-01 04:12:14 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:12:35 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:12:36 EST [ISP.0090.0003C] DELIVERY STATUS 0032702331
2007-08-01 04:12:36 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:13:01 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:13:45 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 04:26:56 EST [ISP.0090.0003C] portalUser = tongren-boyu
2007-08-01 05:30:23 EST [ISP.0090.0003C] portalUser = yz-zhongxing-sjl
2007-08-01 05:49:12 EST [ISS.0025.0001C] Integration Server 6.1 Build 132
2007-08-01 05:49:12 EST [ISS.0025.0006C] License Manager started
2007-08-01 05:49:12 EST [ISS.0025.0041C] FIPS mode not initialized
2007-08-01 05:49:13 EST [BAS.0123.0001C] FSData full consistency check is being performed. Db:WmRepository4/FSD
2007-08-01 05:49:14 EST [ISS.0025.0017C] Repository Manager started
2007-08-01 05:49:20 EST [ISS.0025.0024C] JDBC Connection Manager started
.
.
.
.
.

2

Did you reported this issue to your wM Admin or SysAdmin to troubleshoot on the problem as this is a Prod system?? When is your IS last started with no problems??? Are you worried about the Exceptions that logged above??

HTH,
RMG

3

The log doesn’t look like a restart. Please notedown the PID process Id of the IS on the server. When you think it restarted, check the PID again. If this is external, you should see different PIDs(i think a restart will retain the same PID).

May be your IS is dying and getting started by an external application/human.

4

We have reported about the issue. Worry is that running transactions are dying off coz of server restart.

This restart happened on 1st Aug first, than on 8th Aug…The server log is very much similar on 8th aug also,as we neither find an exception(which could be the cause of restart) nor excessive thread usage warning in server log, but seeing stats log thread usage went upto maximum limit at the time of restart i.e. 400. But again the question is, servers are in cluster, other server was quite stable at the time, why the load is not shared with other server?

5
6

Hi, I couldnt really check on PID as this restart occured on 1st Aug but I dont think that anybody has started this server manually coz nobody maintains this server other than me.