Unable to restart the Integration Server

sunil_kumar10 · June 6, 2017, 6:02pm

Hi All,

Please anybody can help that I have shutdown the Integration Server. And now I not able to start the Integration Server.

Please find the below log that I have taken from the nohup.out log

Changing working directory to /opt/webMethods822/IntegrationServer
Starting as an application using Java because it is not set to use Tanuki Wrapper.
Debug options:
file:/opt/webMethods822/profiles/IS/bin/.options loaded
Registering Software AG Platform Hooks …
log4j:WARN No appenders could be found for logger (com.softwareag.platform.jaas.internal.Activator).
log4j:WARN Please initialize the log4j system properly.
[Journal Logger Initializer] INFO: Reading config root “com.softwareag.platform.logging.configroot”
[Journal Logger Initializer] INFO: Resolved config root to “com.softwareag.platform.logging.configroot”.
06-Jun-2017 16:56:25 com.webMethods.sc.config.ConfigurationLogger log
INFO: Out-of-band file changes will be monitored every ‘60000’ milliseconds
Persistence bundle starting…
Persistence bundle started.
06-Jun-2017 16:56:26 Log InfoDeviceManager started
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to add.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No IU properties to add.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to remove.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No IU properties to remove.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No installable units to move.
[p2] Tue Jun 06 16:56:27 BST 2017 - [Start Level Event Dispatcher] [reconciler] No extra requirements.
Starting Integration Server
06-Jun-2017 16:56:57 com.webMethods.sc.config.ConfigurationLogger log
INFO: Out-of-band file changes will be monitored every ‘60000’ milliseconds
!SESSION 2017-06-06 16:56:23.708 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.6.0_27
java.vendor=Sun Microsystems Inc.
BootLoader constants: OS=solaris, ARCH=sparc, WS=gtk, NL=en_GB

!ENTRY org.eclipse.osgi 2 0 2017-06-06 16:57:00.678
!MESSAGE Info: cycle(s) found while stopping bundles: [[com.ctc.wstx,javax.xml.stream]].
Persistence bundle stopping…
Persistence bundle stopped.
[p2] Tue Jun 06 16:57:01 BST 2017 - [System Bundle Shutdown] Unsaved preferences when shutting down org.eclipse.equinox.internal.p2.metadata.repository.MetadataRepositoryManager

Regards,
Sunil

Holger_von_Thomsen · June 6, 2017, 7:22pm

Hi Sunil,

can you provide the content of /opt/webMethods822/IntegrationServer/logs/server.log?

The nohup.out is not pointing in the right direction currently.

Make sure that IS has been shutdown completely before restarting.
Sometimes shutdown is not completing.

What was the reason for shutting down the IntegrationServer?

Before restarting the next time rename the nohup.out and server.log so you will have a fresh one which just contains the informations about this restart try.

Regards,
Holger

sunil_kumar10 · June 9, 2017, 1:30am

Hi Holger,

I have attached the clean log of restart of application.

And
What was the reason for shutting down the IntegrationServer?
→ we have used update manager to upgrade the patch and after server went to down and going to be restart as showing in the command line and IS did not come to up.
and the moment I am trying to restart the server, I have found that system IS came up and went to down.

Could you please look into the log and suggest what we can do to start the server.

Regards,
Sunil
server.log (8.53 KB)

Holger_von_Thomsen · June 9, 2017, 1:21pm

Hi Sunil,

please restore the config files from config/backup to config, esp. those from the jdbc directory structure.
Try to start the IntegrationServer.

If it still fails try the following:

Backup the following directories:

IntegrationServer/db
IntegrationServer/DocumentStore
IntegrationServer/XAStore
IntegrationServer/WmRepository4

Remove the following directories:

IntegrationServer/db
IntegrationServer/DocumentStore
IntegrationServer/XAStore

Try to start again.

Please provide us the outcome of this.

Regards,
Holger

sunil_kumar10 · June 13, 2017, 3:36pm

Hi Holger,

Sorry for the late response.

I have taken backup of folders that you said. And I tried start the application again but application is still down. I have attached the log for your reference.

Regards,
Sunil
server.log (36.5 KB)

sunil_kumar10 · June 13, 2017, 3:43pm

Hi Holger,

Please find the server log again that has reporting extra information comparing to the last log that I have attached.

Regards,
Sunil

server.log (54.9 KB)

Holger_von_Thomsen · June 13, 2017, 4:10pm

Hi Sunil,

from the logs I see the following:

SAP-Libraries are outdated:


2017-06-13 14:31:26 BST [SAP.0101.0026C] SAP JCo version: 3.0.12 (2014-11-21) JavaRfc 2.2.13
2017-06-13 14:31:26 BST [SAP.0101.0027C] SAP JCo Native Library version: 720.713
2017-06-13 14:31:26 BST [SAP.0101.0044C] SAP IDoc Library version: 3.0.11 (2014-06-20)

Recent versions are JCo 3.0.16 and IDoc Library 3.0.12.

Please check your Broker if it is up and running and that it is configured correctly.
Broker should started BEFORE starting the IntegrationServer.
Please check connectivity to your BEA/Oracle WebLogic Server.
Did you apply some IS_Core Fixes lately?
If so, you will have to apply DC_DBS_Fix and update your ISInternal Database Component by using the Database Configurator.
This is related to the following message:
[ISS.0137.0017E] Scheduler: Exception getting tasks from database: [sag-cjdbc42-0011][Oracle JDBC Driver][Oracle]ORA-00904: “RUN_AT”: invalid identifier
Is there another IS running on the same box?


2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 9999
2017-06-13 14:37:53 BST [ISS.0070.0013W] Failed to start listener HTTPListener@9999. The following error was encountered: Address already in use
2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 5555
2017-06-13 14:37:53 BST [ISP.0046.0012I] Enabling HTTP Listener on port 15006
2017-06-13 14:37:53 BST [ISS.0070.0013W] Failed to start listener HTTPListener@15006. The following error was encountered: Address already in use

If so, please change the Diagnostic and PRT Port.
This can be found in the following files:
IntegrationServer/config/server.cnf
IntgerationServer/packages/WmRoot/config/listeners.cnf
IntgerationServer/packages/WmPRT/config/listeners.cnf

Can you provide a list of installed Fixes from UpdateManager?

Regards,
Holger

sunil_kumar10 · June 13, 2017, 4:39pm

Hi Holger,

Please find the applied patches:

Installed? Fix Name

```
[I] All fixes
```
```
[I]   Adapter Runtime
```
```
[I]     Adapter Runtime 8.2 SP2 Fix 7
```
```
[I]   Adapters
```
```
[I]     SAP Adapter 7.1 Fix13
```
```
[I]     JDBC Adapter 6.5 Fix 33
```

[I]     webMethods JMS Adapter 6.1 Fix 16

```
[I]   Broker
```
```
[I]     Broker_8.2.2_CSharpApi_Fix1
```
[I] Broker_8.2.2_Core_Fix2 (SOL)
[I] Broker_8.2.2_JMSApi_Fix1
[I] Common Library
[I] SCG_8.2SP2_Audit_Fix4
[I] SCG_8.2.2_Terracotta_Fix4
[I] ABE_CL_8.2_SP2_Fix11
[I] SCG_8.2_SP2_Entrust_Fix4
[I] SCG_8.2_SP2_Glue_Fix2
[I] Database Configuration
[I] DC_8.2_SP2_DBS_Fix15
[I] Integration Server
[I] DEP_8.2_SP2_Fix3
[I] Integration Server Core 8.2 Fix 23
[I] Flat File 8.2 SP2 Fix6
[I] IS_8.2_SP2_Tomcat6_Fix6
[I] Monitor UI
[I] Monitor MWS 8.2 SP1 Fix 6
[I] My webMethods Server
[I] MWS_8.2_SP1_Fix7
[I] Broker_8.2.1_Portal_Fix2
[I] SSX
[I] Fix 8.2.2.0003-0833 for SSX (SOL)
[I] Web Services Stack
[I] WSStack 8.2 SP2 Fix19
[I] webMethods Monitor
[I] Monitor 8.2.1 Fix 6

Regards,
Sunil

Holger_von_Thomsen · June 13, 2017, 6:50pm

Hi Sunil,

did you run the database configurator after applying the DC_8.2_SP2_DBS_Fix15 to your installation?

As per the Readmes you are htting the issue PIE-36477.


PIE-36477 (IS_8.2_SP2_Core_Fix20)
After transitioning to or from daylight savings time, scheduled 
tasks run twice or not at all. 

Integration Server runs scheduled tasks based on the time relative 
to the time zone. This caused issues when entering and exiting 
daylight savings time, specifically tasks ran twice or not at all. 

Now, Integration Server runs scheduled tasks based on coordinated 
universal time (UTC). Because Integration Server runs the tasks 
without regard to the relative time zone, the start and end of 
daylight savings time does not affect the execution of scheduled tasks. 

Important! If you are using an external database for the ISInternal 
functional alias in Integration Server, after installing a fix that 
includes PIE-36477 and before starting Integration Server, make sure 
that the latest scripts for ISInternal have been applied to the 
database used by the JDBC pool alias associated with the 
ISInternal functional alias.

Can you check your database schema for the view INSTALLED_COMPONENT (only lists latest version) or table COMPONENT_EVENT (lists all versions incl. migrations to newer versions) for the versions of the different database parts?

From your list of Fixes I miss the SCG_8.2_SP2_DataDirect4.2_Fix1.
But your log files indicate that you are running a DataDirect Driver 4.2 already.

You should check if it is possible for you to apply all latest Fixes for 8.2 to your environment as this will be neccessary anyway when planning a migration to a more recent version of webMethods.

May I suggest to migrate from JMS Adapter to the Built-In JMS services pub.jms:*?
This should work even with BEA/Oracle WebLogic Servers.

Regards,
Holger

sunil_kumar10 · June 14, 2017, 2:07pm

Hi Holger,

The system Integration server came up. But the diagnostic port is showing as disabled and not able to enable the port and it is showing as:

Failed to start HTTPListener@9999: Address already in use

And schedulers also not displaying and throwing error as

[SoftwareAG][Oracle JDBC Driver][Oracle]ORA-00904: “RUN_AT”: invalid identifier

Could you please help on this as well.

Regards,
Sunil

Holger_von_Thomsen · June 14, 2017, 4:24pm

Hi Sunil,

please read my previous post.

It has the informations for the RUN_AT issue (PIE-36477).

Do you have Optimize installed on the same box?
If so, shutdown InfrastructureDC as it is IntegrationServer-based.

Additionally you can change the Diagnostic Port to any other free port between 1025 and 65535.

Regards,
Holger

sunil_kumar10 · June 15, 2017, 4:47pm

Hi Holger,

As you have told I have run the dbConfigurator with migrate and recreate actions.

Before running script the catalogue is:

```
 Installed Components      *
```

BLZ 10 Blaze WEBMDATA,WEBMINDX
RUL 10 BusinessRules WEBMDATA,WEBMINDX
CCS 25 Central Configuration WEBMDATA,WEBMINDX
XRF 11 CrossReference WEBMDATA,WEBMINDX
DSL 20 DistributedLocking WEBMDATA,WEBMINDX
IDR 10 DocumentHistory WEBMDATA,WEBMINDX
ISC 41 ISCoreAudit WEBMDATA,WEBMINDX
ISI 40 ISInternal WEBMDATA,WEBMINDX
MED 10 MediatorEvents WEBMDATA,WEBMINDX
MWS 36 My webMethods Server WEBMDATA,WEBMINDX
OPM 24 Operation Management WEBMDATA,WEBMINDX
PRA 50 ProcessAudit WEBMDATA,WEBMINDX
PRE 60 ProcessEngine WEBMDATA,WEBMINDX
PRP 25 Reporting WEBMDATA,WEBMINDX
PST 20 Staging WEBMDATA,WEBMINDX
SRP 15 SuiteReporting WEBMDATA,WEBMINDX

and After it is showing as:

Executing action
Action : catalog
Database : Oracle
URL : jdbc:wm:oracle://10.143.1.122:1630;serviceName=WMDB
User : wmadmin

```
 Installed Components      *
```

ACT 16 ActiveTransfer WEBMDATA,WEBMINDX
CCS 25 Central Configuration WEBMDATA,WEBMINDX
ISC 41 ISCoreAudit WEBMDATA,WEBMINDX
ISI 40 ISInternal WEBMDATA,WEBMINDX
OPM 24 Operation Management WEBMDATA,WEBMINDX
PRA 50 ProcessAudit WEBMDATA,WEBMINDX
PRE 60 ProcessEngine WEBMDATA,WEBMINDX

And I have gone through the tables and it seems some tables missed over there. Is it will make any problem and please suggest what all actions we have to perform.

Regards,
Sunil

Holger_von_Thomsen · June 19, 2017, 1:48pm

Hi Sunil,

please can you explain for which components/products you have run the dbConfigurator?

Except for very rare cases it should not be neccessary to recreate any parts of the database schema.

I use separate schemas for different parts of the databases:

MWS
Trading Networks
Optimize
Archive (IS and BPM)
IntegrationServer (IS, BPM, Reporting, Staging)
…

This eases to determine which parts of database are affected.

When the tables for the MWS are lost your MWS should no longer be working.

Regards,
Holger

sunil_kumar10 · June 28, 2017, 3:03pm

Hi Holger,

I have run the dbConfigurator for all schemas. Because after migrating schema also issue is still persisting.

And your words are correct that is MWS is not loading.

What is the next action to perform to get the tables that support MWS.

And now I am getting the new error as below for the schedulers:

[SoftwareAG][Oracle JDBC Driver][Oracle]ORA-00604: error occurred at recursive SQL level 1 ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM ORA-02002: error while writing to audit trail ORA-00604: error occurred at recursive SQL level 1 ORA-01653: unable to extend table SYS.AUD$ by 8192 in tablespace SYSTEM

Please help.

Regards,
Sunil

Holger_von_Thomsen · July 5, 2017, 6:33pm

Hi Sunil,

please contact your DBA to enhance quota settings or filesize for Tablespace SYS.

Auditlogging inside database has exceeded the available space in the tablespace file.

Regards,
Holger