9.6 IS is down and not coming up

, ,
1

Hi,

our 9.6 Integration server is down and its not coming up after several normal restarts.
I tried to restart it in safe mode by executing the command: ./console.sh -safemode -debug Debug -port 8731

its failing with below error: and i have no idea where its going wrong. Can somebody please help.

[wmadm@ux311v005 bin]$ ./console.sh -safeboot -debug Debug -port 8731
Starting as an application using Java because it is not set to use Tanuki Wrapper.
Java HotSpot™ 64-Bit Server VM warning: ignoring option MaxPermSize=128M; support was removed in 8.0
Configuration location:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/
Configuration file:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/config.ini loaded
Install location:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/
Framework located:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/org.eclipse.osgi_3.9.0.v20130529-1710.jar
Loading extension: reference:file:com.softwareag.platform.hook_9.6.0.0000-0290.jar
eclipse.properties not found
Loading extension: reference:file:com.softwareag.tps.extra.jre.packages_9.6.0.0000-0011.jar
eclipse.properties not found
Loading extension: reference:file:org.eclipse.equinox.transforms.hook_1.0.200.v20100503.jar
eclipse.properties not found
Loading extension: reference:file:org.glassfish.main.transaction.javax.transaction_3.1.2.1-SNAPSHOT.jar
eclipse.properties not found
Framework classpath:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/org.eclipse.osgi_3.9.0.v20130529-1710.jar
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/common/runtime/agent/repository/plugins/
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/com.softwareag.platform.hook_9.6.0.0000-0290.jar
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/com.softwareag.tps.extra.jre.packages_9.6.0.0000-0011.jar
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/org.eclipse.equinox.transforms.hook_1.0.200.v20100503.jar
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/…/…/common/runtime/agent/repository/plugins/org.glassfish.main.transaction.javax.transaction_3.1.2.1-SNAPSHOT.jar
Debug options:
file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/bin/.options loaded
Registering Software AG Platform Hooks …
Time to load bundles: 7
Oct 30, 2018 3:19:16 PM com.softwareag.platform.shutdown.internal.Activator start
INFO: Shutdown port listener disabled: Required property “com.softwareag.platform.shutdown.port” is not set or empty.
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: osgi.user.area=file:/home/wmadm/user/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: osgi.instance.area=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/workspace/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: osgi.configuration.area=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: osgi.install.area=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: eclipse.home.location=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator replaceSystemPropertyLocations
INFO: Replacing system property @url.java.security.auth.login.config->@osgi.configuration.area:jaas.config with java.security.auth.login.config->file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/jaas.config
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator replaceSystemPropertyLocations
INFO: Replacing system property @url.sag.install.area->@osgi.install.area:…/… with sag.install.area->file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator replaceSystemPropertyLocations
INFO: Replacing system property @path.catalina.base->@osgi.configuration.area:tomcat with catalina.base->/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/tomcat
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator replaceSystemPropertyLocations
INFO: Replacing system property @path.tomcat.home->@osgi.configuration.area:tomcat with tomcat.home->/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/tomcat
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.Activator$1 addingService
INFO: Discovered location: sag.install.area=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.startup.internal.LocationRegistry open
INFO: Registered location sag.install.area=file:/opt/webMethods/v9_6/ISB/MED/INTERNAL/
Oct 30, 2018 3:19:16 PM com.softwareag.platform.console.internal.LocalConsoleManager createConsole
INFO: Creating a local JLine console over terminal: jline.NoInterruptUnixTerminal@4661c619
Oct 30, 2018 3:19:16 PM com.softwareag.platform.console.internal.LocalConsoleManager$1 run
INFO: Starting a local JLine console sesssion
osgi$ Oct 30, 2018 3:19:16 PM com.softwareag.platform.logging.init.internal.Activator start
INFO: Setting configuration directory to /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration
Oct 30, 2018 3:19:17 PM com.webMethods.sc.config.ConfigurationLogger log
INFO: Out-of-band file changes will be monitored every ‘60000’ milliseconds
log4j:WARN Unrecognized element threshold
Starting Integration Server
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/bin/lib/wm-isserver.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-isclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/sin-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/castor-all.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/eda-eventtypes.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/eda-orchestrator.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ehcache-core-ee.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ehcache-terracotta-ee.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glue.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/saglic.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/terracotta-toolkit-runtime-ee.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-acdl-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-bpm-processmodel.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-client.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-event-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-jsf.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-rules.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-caf-server.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-calendar-components.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-calendar.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-cdc-core.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-directory-components.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-directory.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-eform-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-eform.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-g11nutils.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-lwq.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-metadata-core-centrasite.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-metadata-core.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-mws-library.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-scg-audit.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-scg-cluster.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-scg-core.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-scg-jdbcpool.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-scg-security.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-trang.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-wsclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wsdl4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wss4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/xmlsec.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-broker-acdl.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-brokerclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-jmsclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wm-jmsnaming.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteCommons.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteDynLoader.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteDynLoaderExt.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteFramework-L10N.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteFramework-L10N_en.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteFramework-common.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteImportExport.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteJAXR-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteLCM-L10N.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteLCM-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteLCM.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSitePolicy-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteResourceAccess-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteUtils-L10N.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteUtils.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteVMS-L10N.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/CentraSiteVMS.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/PolicyLogBindings.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/TaminoAPI4J-l10n.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/TaminoAPI4J.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/cstUtils.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/INMConfiguration.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/INMConfiguration-L10N.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/inmUtil.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/uddiKeyConverter.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/uddiv3ClientAPI.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/wvcm.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/xqj-ino-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/xqjapi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/activation.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/axis.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-beanutils.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-codec.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-collections.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-digester.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-discovery.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-fileupload.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-httpclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-io.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-lang.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/commons-logging.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/dd-cjdbc.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/derby.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/enttoolkit.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/esapi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/httpclient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/httpcore.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/ical4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/icu4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jackrabbit-jcr-rmi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jakarta-oro.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jaxen.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jcr.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jdom.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/joda-time.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jsch.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jsp-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jsr173_api.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/jzlib.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/ldapfilt.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/ldapjdk.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/log4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/opensaml.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/opensaml2.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/openws.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/portlet-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/relaxngDatatype.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/saxpath.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/serializer.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/servlet-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/slf4j-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/slf4j-log4j.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/stax-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/velocity.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/wstx-asl.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/xalan.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/xercesImpl.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/xmlbeans.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/xmltooling.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.jackson-core-asl.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.javax.jms.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.javax.mail.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.javax.resource.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.javax.transaction.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.jaxb-API.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.jaxb-osgi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.jaxr-API-osgi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/glassfish/gf.jaxrpc-API-osgi.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/nAdminAPI.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/nAdminXMLAPI.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/nClient.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/nJMS.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/nP2P.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/bam-ws-intf.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/bam-ws-client.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/i2z_clientapi_withdeps.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/lib/ext/CspServerDeps.jar
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/common/conf
Classpath entry in ini.cnf not found: /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/WS-Stack/lib
Exception launching ServerApp
com.wm.proxy.ProxyException: failed to invoke proxy method
com.wm.app.b2b.server.Main

    at com.wm.proxy.Proxy.invokeStaticMethod(Proxy.java:164)
    at com.wm.app.server.ServerProxy.ServerMain_main(ServerProxy.java:64)
    at com.wm.app.server.ServerApp.start(ServerApp.java:91)
    at com.wm.app.Application.launch(Application.java:638)
    at com.softwareag.osgi.is.launcher.ISLauncher$ServerThread.run(ISLauncher.java:195)
    at java.lang.Thread.run(Thread.java:745)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

Caused by: java.lang.ClassNotFoundException: com.wm.app.b2b.server.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:344)
at com.wm.proxy.Proxy.invokeStaticMethod(Proxy.java:159)
… 8 more
!SESSION 2018-10-30 15:19:15.845 -----------------------------------------------
eclipse.buildId=unknown
java.version=1.8.0_31
java.vendor=Oracle Corporation
BootLoader constants: OS=linux, ARCH=x86_64, WS=gtk, NL=en_US
Framework arguments: -safeboot -port 8731
Command-line arguments: -safeboot -debug Debug -port 8731

!ENTRY com.softwareag.platform.hook 0 0 2018-10-30 15:19:17.767
!MESSAGE Loaded class ‘org.apache.camel.Component’ from ProxyService: 170

!ENTRY com.softwareag.platform.hook 0 0 2018-10-30 15:19:17.770
!MESSAGE Loaded class ‘org.apache.camel.TypeConverter’ from ProxyService: 170
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] User requested forced reconciliation via “osgi.checkConfiguration=true” System property.
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] Performing reconciliation.
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] User requested forced reconciliation via “osgi.checkConfiguration=true” System property.
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] Performing reconciliation.
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] Nothing to do.
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] Writing out timestamps to file : /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/org.eclipse.osgi/bundles/217/data/cache.timestamps
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/configuration/org.eclipse.update/platform.xml=0
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/dropins=1414145465000
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/features=0
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/links=0
[p2] Tue Oct 30 15:19:18 CET 2018 - [Start Level Event Dispatcher] [reconciler] /opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/plugins=1470831876000

!ENTRY org.eclipse.equinox.cm 4 0 2018-10-30 15:19:21.871
!MESSAGE Failed to create bean jmxConnector
!STACK 0
java.lang.RuntimeException: Failed to create bean jmxConnector
at com.softwareag.platform.blueprint.ca.internal.AbstractManagedService.create(AbstractManagedService.java:235)
at com.softwareag.platform.blueprint.ca.ManagedServiceFactory.updated(ManagedServiceFactory.java:49)
at org.eclipse.equinox.internal.cm.ManagedServiceFactoryTracker$2.run(ManagedServiceFactoryTracker.java:190)
at org.eclipse.equinox.internal.cm.SerializedTaskQueue$1.run(SerializedTaskQueue.java:36)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at com.softwareag.platform.blueprint.ca.internal.BeanLifecycle.create(BeanLifecycle.java:143)
at com.softwareag.platform.blueprint.ca.internal.BeanLifecycle.open(BeanLifecycle.java:78)
at com.softwareag.platform.blueprint.ca.internal.AbstractManagedService.create(AbstractManagedService.java:233)
… 3 more
Caused by: java.io.IOException: JMX Server on 0.0.0.0:8075 failed to create an RMI registry RMIServerSocketFactory
at com.softwareag.platform.jmx.internal.JmxConnectorImpl.createRegistry(JmxConnectorImpl.java:223)
at com.softwareag.platform.jmx.internal.JmxConnectorImpl.openImpl(JmxConnectorImpl.java:150)
at com.softwareag.platform.jmx.internal.JmxConnectorImpl.open(JmxConnectorImpl.java:113)
… 10 more
Caused by: java.rmi.server.ExportException: Port already in use: 8075; nested exception is:
java.net.BindException: Address already in use
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:341)
at sun.rmi.transport.tcp.TCPTransport.exportObject(TCPTransport.java:249)
at sun.rmi.transport.tcp.TCPEndpoint.exportObject(TCPEndpoint.java:411)
at sun.rmi.transport.LiveRef.exportObject(LiveRef.java:147)
at sun.rmi.server.UnicastServerRef.exportObject(UnicastServerRef.java:208)
at sun.rmi.registry.RegistryImpl.setup(RegistryImpl.java:152)
at sun.rmi.registry.RegistryImpl.(RegistryImpl.java:112)
at java.rmi.registry.LocateRegistry.createRegistry(LocateRegistry.java:239)
at com.softwareag.platform.jmx.internal.JmxConnectorImpl.createRegistry(JmxConnectorImpl.java:220)
… 12 more
Caused by: java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.AbstractPlainSocketImpl.bind(AbstractPlainSocketImpl.java:382)
at java.net.ServerSocket.bind(ServerSocket.java:375)
at java.net.ServerSocket.(ServerSocket.java:237)
at javax.net.DefaultServerSocketFactory.createServerSocket(ServerSocketFactory.java:231)
at com.softwareag.platform.jmx.internal.PlainRMIServerSocketFactory.createServerSocket(PlainRMIServerSocketFactory.java:18)
at sun.rmi.transport.tcp.TCPEndpoint.newServerSocket(TCPEndpoint.java:666)
at sun.rmi.transport.tcp.TCPTransport.listen(TCPTransport.java:330)
… 20 more

2

Hi Jyoti,

is there anything under /opt/webMethods/v9_6/ISB/MED/INTERNAL/IntegrationServer/instances/default/logs/server.log?

This might indicate where IS is failing startup.

Please check if you can replace ini.cnf with ini.cnf_bak as it looks that it got corrupted.

Regards,
Holger

3

Also check wrapper.log under profiles.

4

Hi Holger, Mahesh

I tried to replace the ini.cnf file from backup and tried the command: ./console.sh -safeboot -debug Debug -port 8731, but same error i got.
Please find the server.log attached. There are some errors with the keystore files in the log, and some other errors also, but am not sure how to resolve that.
Could you please have a look.
I could not find the wrapper.log in the profiles folder.

Regards
Jyoti

5

Sorry, my bad…forgot to attach the file.

server.log (16.5 KB)

6

Hi Jyoti,

From the server log, we can see JDBC pool is not correctly configured.

Can you please validate and configured ISInternal JDBC pool properly start the server

Thanks,
Yogesh

7

Hi Jyoti,

please check your database pool definitons under IntegrationServer/instances/default/config/jdbc/pool first.

There seem to be 4 definitions, from which 2 are defining the same Alias name.

From Server.log:
2018-10-30 12:58:01 CET [ISS.0096.0005C] JDBCConnectionManager: initialization failed with exception [BAJ.0001.0003] JDBCConnMgr: Duplicate Connection Pool alias ISB_MED_INT_CORE
2018-10-30 12:58:01 CET [ISS.0096.0005C] JDBCConnectionManager: initialization failed with exception [BAJ.0001.0003] JDBCConnMgr: Duplicate Connection Pool alias ISB_MON

After this is fixed check as Yogesh has described.
Especially fix the jdbc pool assignment for ISInternal function as this is required when using Scheduled Tasks.

The errors with the keystores indicate that IS is not able to find the files at the specified location.
Were they moved, renamed or deleted (by accident)?

Regards,
Holger

8

Hi Holger, Yogesh,

Yes, there were 2 backup files of the JDBC pool files located in the same directory with same names. Removed those backup files.
I will drop and recreate the ISInternal functional alias using dcConfigurator and update here.

And regarding the keystore files, i can see the keystore file in that location /home/wmadm/.cert/ux311v005.jks, and those are not modified since July 2017.

-rw-r–r–. 1 wmadm wmadm 10982 Jul 24 2017 ux311v005.jks

Regards
Jyoti

9

Hi Jyoti,

Regarding keystore error, please make sure the password given in keystore is same as in IS.

Thanks,
Yogesh

1 Like
10

Hi Jyoti,

I dont think that it is neccessary to recreate the database part here.

Just fix the pool settings (the alias naming) and the assignmnent between function and pool.
After that test all of these and restart the functions.

Regards,
Holger

11

Hi,

I deleted the duplicate pool aliases and recreated the ISInternal, CentralUser and Xref functional aliases.
But am still seeing the same errors.
But i have been getting this error everytime, can you please tell me from which file its taking db details:

Error creating database connection to: jdbc:wm:oracle://ux302v101:1521;ServiceName=vwmcom3_appl;AlternateServers=(ux302v102:1521);LoadBalancing=false;catalogOptions=1com.wm.dd.jdbc.oraclebase.ddej: [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-01017: invalid username/password; logon denied

And exactly after this, am getting this:

2018-11-01 13:14:07 CET [CommonLib.MWS.0002W] [Pool - default] is full. waiting for checkin: [Pool - default]: MaxSize = 256; total pool size = 0; idle pool size = 0; check out size = 0
2018-11-01 13:14:07 CET [CommonLib.MWS.0002I] No registered exception delegate
2018-11-01 13:14:07 CET [CommonLib.MWS.0002I] webMethods Integration Server Shutting down…
2018-11-01 13:14:07 CET [CommonLib.MWS.0002I] StackTrace for shutdown - this is an expect info message only :
java.lang.Thread.getStackTrace(Thread.java:1552)
com.webMethods.portal.system.PortalSystem.shutdown(PortalSystem.java:950)
com.webMethods.portal.system.PortalSystem.main(PortalSystem.java:870)
com.webMethods.sc.mws.MWSLibrary.init(MWSLibrary.java:271)
com.wm.app.b2b.server.cds.CDSUserManager.initialize(CDSUserManager.java:186)
com.wm.app.b2b.server.UserManager.jndiInit(UserManager.java:557)
com.wm.app.b2b.server.UserManager.init(UserManager.java:488)
com.wm.app.b2b.server.Server.run(Server.java:398)

Please help.

Regards
Jyoti

12

Also, one more thing that i noticed is when i go for shutdown of the server, it shows its already running. Then, why am not able to access it.

[wmadm@ux311v005 bin]$ ./shutdown.sh
cp: writing `/opt/webMethods/v9_6/ISB/MED/INTERNAL/profiles/IS_default/shutdown.anchor’: No space left on device
Checking has started…
Software AG Integration Server 9.6 (default) is running: PID:27391, Wrapper:STARTED, Java:STARTED

13

Hi,

please check your file system where IS is installed as there is not enough space on it.

Either remove some files from your FS or get it increased (if possible) by your Unix Admin.

After that shutdown and restart your IS and try to reconfigure the JDBC pools in IS Admin UI.

When there is not enough space on the FS this might lead to corruption of configuration files rendering the IS being unable to start.
Under IntegrationServer/instances/default/config/backup is always a copy of the last successful start attempt, which can be copied back to IntegrationServer/instances/default/config/ whenever one of these files get corrupted.

The start and stop scripts do not check if the IS is fully started, but only if there is a corresponding JVM process running.
That´s why it states that it is already running, but is not accesible by Admin UI.

Regards,
Holger

14

Hi Holger,

That was a very good point, and as i checked, i could see the size was full.

[wmadm@ux311v005 /]$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/appvg-webMethodslv
127G 121G 18M 100% /opt/webMethods

But, i had checked the size on Tuesday, and it was almost 60% available. Please see below.

/dev/mapper/appvg-webMethodslv
127G 51G 71G 42% /opt/webMethods

i am curious how is it filling up so fast. Also, if i clean up or ask the unix team to increase the size bit more, how can i be sure this wont come up again.
Could you please help me understand.

Regards
Jyoti

15

Hi Jyoti,

you will have to find the files or directories which are piling up.
Get assistance from OS team if needed.

Check the logging configuration and log files in IS and reduce the log level when neccessary.

Generally logging as DEBUG produces a quite huge server.log, i.e. a lot of HTTP Request data when accessing IS Admin UI.
Set the log level only for those factories you are interested in.

Regards,
Holger

16

Thank you Holger.
I will check the size issue and update the result.
Actually we have a clustered node in this case. The available space in the other server is quite good.

/dev/mapper/appvg-webMethodslv
128G 58G 64G 48% /opt/webMethods

And the JDBC pools in both the servers are connecting to the same DB with same schema.

The other server is also down, and when i try to bring it up, am getting the below error:

2018-11-02 11:34:48 CET [ISS.0025.0001C] Integration Server 9.6.0.0 Build 294
2018-11-02 11:34:48 CET [ISS.0025.0006I] License Manager started
2018-11-02 11:34:48 CET [ISS.0001.0008W] Publish / Subscribe Functionality disabled due to license restrictions.
2018-11-02 11:34:48 CET [ISS.0001.0008W] Adapter Runtime Functionality disabled due to license restrictions.
2018-11-02 11:34:48 CET [ISS.0001.0008W] Remote Invoke Functionality disabled due to license restrictions.
2018-11-02 11:34:48 CET [ISS.0001.0008W] Guaranteed Delivery Functionality disabled due to license restrictions.
2018-11-02 11:34:48 CET [ISS.0001.0008W] Distributed Cache Functionality disabled due to license restrictions.
2018-11-02 11:34:49 CET [ISS.0025.0050W] The JCE Unlimited Strength Jurisdiction Policy File was not found. Please install it.
2018-11-02 11:34:50 CET [ISS.0025.0041I] FIPS mode not initialized
2018-11-02 11:34:51 CET [BAS.0123.0001I] FSData full consistency check is being performed. Db:WmRepository4/RepoV4
2018-11-02 11:34:52 CET [ISS.0025.0017I] Repository Manager started
2018-11-02 11:34:54 CET [ISS.0096.0001C] JDBCConnectionManager: initialization failed for CentralUsers functional alias, check configuration.
2018-11-02 11:34:55 CET [ISS.0096.0001C] JDBCConnectionManager: initialization failed for ISInternal functional alias, check configuration.
2018-11-02 11:34:59 CET [ISS.0025.0024I] JDBC Connection Manager started
2018-11-02 11:34:59 CET [ISS.0095.0015W] Database at URI java:webMethods/jdbc/ProcessAudit is not available.
2018-11-02 11:34:59 CET [ISS.0095.0018W] Cannot change destination of logger Process Logger to file system.
2018-11-02 11:34:59 CET [ISS.0095.0019W] Disabling the logger Process Logger.
2018-11-02 11:34:59 CET [ISS.0095.0015W] Database at URI java:webMethods/jdbc/ISCoreAudit is not available.
2018-11-02 11:34:59 CET [ISS.0095.0018W] Cannot change destination of logger Document Logger to file system.
2018-11-02 11:34:59 CET [ISS.0095.0019W] Disabling the logger Document Logger.
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ErrorDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ServiceDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘CoreAuditDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘mediatorTxnJdbc’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘SessionDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ProcessAuditDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ProcessAudit’’
2018-11-02 11:35:00 CET [WmAuditingSC.queue.2019W] Worker thread for queue ‘ProcessAuditQueue’ started, but its target destination is not open
2018-11-02 11:35:00 CET [WmAuditingSC.queue.2019W] Worker thread for queue ‘Document Queue’ started, but its target destination is not open
2018-11-02 11:35:00 CET [ISS.0095.0014W] Audit Logging initialized with errors.
2018-11-02 11:35:00 CET [ISS.0014.0054C] Error occurred while initializing server: com.wm.util.coder.InvalidDatatypeException: [ISC.0076.9002] incorrect IDataXMLCoder file format: no element ‘IDataXMLCoder’ found at com.wm.util.coder.IDataXMLCoder.decode(IDataXMLCoder.java:315) at com.wm.util.coder.IDataXMLCoder.decode(IDataXMLCoder.java:295) at com.wm.util.coder.IDataCoder.readFromFile(IDataCoder.java:108) at com.wm.util.IDataRegistry.readFromFile(IDataRegistry.java:101) at com.wm.util.IDataRegistry.init(IDataRegistry.java:61) at com.wm.util.IDataRegistry.(IDataRegistry.java:43) at com.wm.util.IDataRegistry.(IDataRegistry.java:30) at com.wm.app.b2b.server.sftp.client.SFTPClientManager.(SFTPClientManager.java:102) at com.wm.app.b2b.server.sftp.client.SFTPClientManager.initialize(SFTPClientManager.java:72) at com.wm.app.b2b.server.Server.run(Server.java:388)

Regards
Jyoti

17

Hi Jyoti,

pointing clustered ISes to the same database schema is a requirement for Clustering, that´s fine.

Please check the config-folder for SFTP- and JDBC-config and replace the files with those from the corresponding config/backup location.
Then restart IS.

Regards,
Holger

18

Hi Holger,

As you suggested, i replaced the JDBC and SFTP config files from the respective backup folders, but am getting the below error now: Could you please help.

2018-11-05 10:20:57 CET [ISS.0025.0050W] The JCE Unlimited Strength Jurisdiction Policy File was not found. Please install it.
2018-11-05 10:20:58 CET [ISS.0025.0041I] FIPS mode not initialized
2018-11-05 10:20:59 CET [BAS.0123.0001I] FSData full consistency check is being performed. Db:WmRepository4/RepoV4
2018-11-05 10:21:00 CET [ISS.0025.0017I] Repository Manager started
2018-11-05 10:21:00 CET [ISS.0096.0001C] JDBCConnectionManager: initialization failed for CentralUsers functional alias, check configuration.
2018-11-05 10:21:00 CET [ISS.0096.0001C] JDBCConnectionManager: initialization failed for ISInternal functional alias, check configuration.
2018-11-05 10:21:03 CET [ISS.0025.0024I] JDBC Connection Manager started
2018-11-05 10:21:03 CET [ISS.0095.0015W] Database at URI java:webMethods/jdbc/ProcessAudit is not available.
2018-11-05 10:21:03 CET [ISS.0095.0018W] Cannot change destination of logger Process Logger to file system.
2018-11-05 10:21:03 CET [ISS.0095.0019W] Disabling the logger Process Logger.
2018-11-05 10:21:06 CET [ISS.0095.0015W] Database at URI java:webMethods/jdbc/ISCoreAudit is not available.
2018-11-05 10:21:06 CET [ISS.0095.0018W] Cannot change destination of logger Document Logger to file system.
2018-11-05 10:21:06 CET [ISS.0095.0019W] Disabling the logger Document Logger.
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ErrorDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ServiceDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘CoreAuditDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘mediatorTxnJdbc’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘SessionDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ISCoreAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.main.5010E] Failed to open destination ‘ProcessAuditDBDest’; reason ‘Failed to lookup DataSource for ‘java:webMethods/jdbc/ProcessAudit’’
2018-11-05 10:21:06 CET [WmAuditingSC.queue.2019W] Worker thread for queue ‘ProcessAuditQueue’ started, but its target destination is not open
2018-11-05 10:21:06 CET [WmAuditingSC.queue.2019W] Worker thread for queue ‘Document Queue’ started, but its target destination is not open
2018-11-05 10:21:06 CET [ISS.0095.0014W] Audit Logging initialized with errors.
2018-11-05 10:21:08 CET [CommonLib.MWS.0002I] Initializing component: com.webMethods.rtl.util.xml.XMLUtil
2018-11-05 10:21:08 CET [CommonLib.MWS.0002I] Loading phase: [phaseID, phaseName] [miePhaseTwo, miePhaseTwo]
2018-11-05 10:21:08 CET [CommonLib.MWS.0002I] Initializing components of: miePhaseTwo
2018-11-05 10:21:08 CET [CommonLib.MWS.0002I] Initializing component: com.webMethods.portal.service.sql.core.ConnectionManager
Error creating database connection to: jdbc:wm:oracle://ux302v101:1521;ServiceName=vwmcom3_appl;AlternateServers=(ux302v102:1521);LoadBalancing=false;catalogOptions=1com.wm.dd.jdbc.oraclebase.ddej: [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-00257: Archiver error. Connect AS SYSDBA only until resolved.

    at com.wm.dd.jdbc.oraclebase.ddcq.b(Unknown Source)
    at com.wm.dd.jdbc.oraclebase.ddcq.a(Unknown Source)
    at com.wm.dd.jdbc.oracle.OracleImplConnection.b(Unknown Source)
    at com.wm.dd.jdbc.oracle.OracleImplConnection._(Unknown Source)
    at com.wm.dd.jdbc.oraclebase.BaseConnection.b(Unknown Source)
    at com.wm.dd.jdbc.oraclebase.BaseConnection.k(Unknown Source)
    at com.wm.dd.jdbc.oraclebase.ddak.run(Unknown Source)

2018-11-05 10:22:48 CET [ISS.0028.0005I] Loading WmAssetPublisher package
2018-11-05 10:22:48 CET [ISS.0028.0005I] Loading WmFlatFile package
2018-11-05 10:22:48 CET [ISS.0028.0005I] Loading WmCloud package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmTomcat package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmMediator package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmVCS package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmPRT package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmART package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmClearCase package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmSubversion package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmARTExtDC package
2018-11-05 10:22:49 CET [ISS.0028.0005I] Loading WmDesigner package
2018-11-05 10:22:51 CET [ISS.0028.0005I] Loading Default package
2018-11-05 10:22:51 CET [ISS.0028.0005I] Loading smg1039 package
2018-11-05 10:22:51 CET [ISS.0028.0005I] Loading WmTaskClient package
2018-11-05 10:22:53 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.event:startEventAgentPort)
2018-11-05 10:22:53 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.net.email:init)
2018-11-05 10:22:53 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.net.ftp:init)
2018-11-05 10:22:53 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.net.ftps:init)
2018-11-05 10:22:53 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.net.http:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.ns:syncToNamespace)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.remote:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.proxy:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.schedule:init)
2018-11-05 10:22:54 CET [ISS.0137.0002W] Scheduler: Unable to get pool for ISInternal
2018-11-05 10:22:54 CET [ISS.0137.0033C] Errors encountered while initializing Scheduler. Scheduled tasks will not run. Check the logs for more information.
2018-11-05 10:22:54 CET [ISS.0028.0013E] Startup service exited with the following error: com.wm.app.b2b.server.ServiceException: java.lang.IllegalStateException: No ISInternal JDBC Pool
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.dispatcher:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.tx:init)
2018-11-05 10:22:54 CET [ISC.0063.0042I] Job Manager not initialized.
2018-11-05 10:22:54 CET [ISS.0028.0013E] Startup service exited with the following error: [ISC.0063.9107] [TxMgr] Service Disabled
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.soap:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.tspace:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.ws:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.oauth:init)
2018-11-05 10:22:54 CET [ISS.0028.0013E] Startup service exited with the following error: com.wm.app.b2b.server.ServiceException: com.wm.app.b2b.server.ServiceException: The ISInternal functional alias is not configured properly on the JDBC Pools page. Assign a valid database pool and restart the Integration Server.
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.security.keystore:createDefaultKeyStoreTrustStoreAliases)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.ntlmclient:init)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmRoot: Startup service (wm.server.sftpclient:updateKexList)
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmPublic: Startup service (pub.storage:startup)
2018-11-05 10:22:54 CET [ISS.0028.0013E] Startup service exited with the following error: com.wm.app.b2b.server.ServiceException: The ISInternal functional alias is not configured properly on the JDBC Pools page. Assign a valid database pool and restart the Integration Server.
2018-11-05 10:22:54 CET [ISS.0028.0012I] WmPRT: Startup service (wm.prt.admin:startup)
2018-11-05 10:22:54 CET [BPM.0101.0521W] Storage cleaner: cannot acquire lock. Exception: [Exception: Unable to connect to database!] not found in “com.wm.resources.CoreExcpMsgs”.
2018-11-05 10:22:54 CET [BPM.0101.1600W] Unable to get Optimize process data collector. Tracking enabled models will not be logged to Optimize.
2018-11-05 10:24:18 CET [CommonLib.MWS.0002W] [Pool - default] is full. waiting for checkin: [Pool - default]: MaxSize = 256; total pool size = 0; idle pool size = 0; check out size = 0
2018-11-05 10:24:18 CET [CommonLib.MWS.0002I] Thread-142 Shutting down…
2018-11-05 10:24:18 CET [CommonLib.MWS.0002I] StackTrace for shutdown - this is an expect info message only :
java.lang.Thread.getStackTrace(Thread.java:1552)
com.webMethods.portal.system.PortalSystem.shutdown(PortalSystem.java:950)
com.webMethods.portal.system.PortalSystem.main(PortalSystem.java:870)
com.webMethods.sc.mws.MWSLibrary.init(MWSLibrary.java:271)
com.wm.app.b2b.server.cds.CDSUserManager.initialize(CDSUserManager.java:186)
com.wm.app.b2b.server.UserManager.jndiInit(UserManager.java:557)
com.wm.app.b2b.server.ACLGroup$GroupLoader$GroupLoadingTask.run(ACLGroup.java:415)
java.lang.Thread.run(Thread.java:745)

2018-11-05 10:24:18 CET [CommonLib.MWS.0002I] Shutting down components of: PortalServer
2018-11-05 10:24:18 CET [CommonLib.MWS.0002I] Shutting down components of: miePhaseOne
2018-11-05 10:24:18 CET [CommonLib.MWS.0002I] Stopped sending JDK log messages to Log4j
2018-11-05 10:24:18 CET [ISS.0024.0010C] Central User Manager initialization failed with error : [POP.004.0200] Cannot create database connection to default MWS datasource.
2018-11-05 10:24:18 CET [ISS.0024.0001E] Could not initialize User Manager.

Regards
Jyoti

19

Hi Jyoti,

please check your database (RAC?) config as well as the URLs in the JDBC pools.
Looks like there is some misconfig somewhere.

I cannot tell much about this as we are not using RAC here, only singled database nodes.

Regards,
Holger