Integration server is down , trying to bringup

, , , , , ,
1

Hi experts,

We got to know our Integration server is down and we tried checking the servers logs, but there is no logs.
I restarted the server, in logs i can see config directory saved and process is running, But IS console is not coming up, in the logs i can see

2022-05-20 01:52:18 PDT [ISS.0014.0025W] Master password for outbound password encryption has expired.
2022-05-20 01:12:15 PDT [ISS.0025.0016I] Config File Directory Saved
2022-05-20 01:13:40 PDT [ISP.0061.0002I] Expired remote connection to ma_b2bt_lapp858662 for user Administrator
2022-05-20 01:13:40 PDT [ISP.0061.0002I] Expired remote connection to ma_b2bt_lapp858662 for user Administrator
2022-05-20 01:13:40 PDT [ISP.0061.0002I] Expired remote connection to ma_b2bt_lapp858662 for user Administrator

Could you please help me to bring back the IS ?

Best Regards
K M,Varun

2

Hi @varunkmv
If you see “Config File Directory Saved” , it means that the server is up,
you can check for the message [ISP.0046.0012I] Enabling HTTP Listener on port num
You can try connecting to the port from different clients like a browser / postman to confirm that the IS is indeed not up.

The logs that you see are either warnings or Informational messages 0025W and 0002I , you can check if there are more logs which clearly show an error. You could also check the /profiles/IS_default/logs wrapper.log files.

-NP

3

ENTRY org.apache.felix.scr 4 0 2022-05-20 02:38:40.130
INFO | jvm 1 | 2022/05/20 02:38:40 | !MESSAGE bundle org.apache.felix.scr:2.1.16 (250)Circular reference detected trying to get service {com.softwareag.platform.bundle.watchdog.BundleWatchdog}={service.id=224, directory=/ngs/app/b2bit/apps/wm105/profiles/IS_default/workspace/app-platform/deployer/bundles, service.bundleid=105, service.scope=bundle, TokenResolver.target=(service.pid=location.token.resolver), component.name=com.softwareag.platform.bundle.watchdog.internal.BundleWatchdogImpl, component.id=12, com.softwareag.config.loader.pid=file.property.loader, service.factoryPid=com.softwareag.bundle.watchdog.pid, com.softwareag.config.source=file:/ngs/app/b2bit/apps/wm105/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader/com.softwareag.bundle.watchdog.pid-AppPlatform.properties, com.softwareag.config.timestamp=1566229524000, service.pid=com.softwareag.bundle.watchdog.pid-1653036522811-10, bundle.watchdog.name=AppPlatform}
INFO | jvm 1 | 2022/05/20 02:38:40 | stack of references: ServiceReference: {com.softwareag.platform.bundle.watchdog.BundleWatchdog}={service.id=224, directory=/ngs/app/b2bit/apps/wm105/profiles/IS_default/workspace/app-platform/deployer/bundles, service.bundleid=105, service.scope=bundle, TokenResolver.target=(service.pid=location.token.resolver), component.name=com.softwareag.platform.bundle.watchdog.internal.BundleWatchdogImpl, component.id=12, com.softwareag.config.loader.pid=file.property.loader, service.factoryPid=com.softwareag.bundle.watchdog.pid, com.softwareag.config.source=file:/ngs/app/b2bit/apps/wm105/profiles/IS_default/configuration/com.softwareag.platform.config.propsloader/com.softwareag.bundle.watchdog.pid-AppPlatform.properties, com.softwareag.config.timestamp=1566229524000, service.pid=com.softwareag.bundle.watchdog.pid-1653036522811-10, bundle.watchdog.name=AppPlatform}
INFO | jvm 1 | 2022/05/20 02:38:40 | ServiceReference: {com.softwareag.platform.bundle.watchdog.internal.FrameworkRefresher}={service.id=223, service.bundleid=105, service.scope=bundle, component.name=com.softwareag.platform.bundle.watchdog.internal.FrameworkRefresher, component.id=14}
INFO | jvm 1 | 2022/05/20 02:38:40 |
INFO | jvm 1 | 2022/05/20 02:38:40 | !STACK 0
INFO | jvm 1 | 2022/05/20 02:38:40 | java.lang.Exception: stack trace
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.ComponentRegistry.enterCreate(ComponentRegistry.java:485)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.BundleComponentActivator.enterCreate(BundleComponentActivator.java:734)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:881)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:215)
INFO | jvm 1 | 2022/05/20 02:38:40 | at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:213)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:114)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:48)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:527)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:464)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:661)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:86)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.inject.BindParameters.getServiceObject(BindParameters.java:47)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.inject.methods.BindMethod.getServiceObject(BindMethod.java:664)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2308)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager$MultipleDynamicCustomizer.prebind(DependencyManager.java:411)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1568)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1029)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:935)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleComponentManager.getService(SingleComponentManager.java:900)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse$1.run(ServiceFactoryUse.java:215)
INFO | jvm 1 | 2022/05/20 02:38:40 | at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.factoryGetService(ServiceFactoryUse.java:213)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceFactoryUse.getService(ServiceFactoryUse.java:114)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceConsumer$2.getService(ServiceConsumer.java:48)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.getService(ServiceRegistrationImpl.java:527)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.getService(ServiceRegistry.java:464)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.getService(BundleContextImpl.java:661)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleRefPair.getServiceObject(SingleRefPair.java:86)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.inject.BindParameters.getServiceObject(BindParameters.java:47)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.inject.methods.BindMethod.getServiceObject(BindMethod.java:664)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager.getServiceObject(DependencyManager.java:2308)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.prebind(DependencyManager.java:1154)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager.prebind(DependencyManager.java:1568)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.collectDependencies(AbstractComponentManager.java:1029)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.SingleComponentManager.getServiceInternal(SingleComponentManager.java:935)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:756)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:1045)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.DependencyManager$SingleStaticCustomizer.addedService(DependencyManager.java:999)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1216)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.customizerAdded(ServiceTracker.java:1137)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.trackAdding(ServiceTracker.java:944)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ServiceTracker$AbstractTracked.track(ServiceTracker.java:880)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:1168)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.BundleComponentActivator$ListenerInfo.serviceChanged(BundleComponentActivator.java:125)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.serviceChanged(FilteredServiceListener.java:112)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.dispatchEvent(BundleContextImpl.java:956)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:233)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchronous(ListenerQueue.java:151)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEventPrivileged(ServiceRegistry.java:865)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServiceEvent(ServiceRegistry.java:804)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.register(ServiceRegistrationImpl.java:130)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerService(ServiceRegistry.java:228)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.registerService(BundleContextImpl.java:506)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:906)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager$3.register(AbstractComponentManager.java:892)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.RegistrationManager.changeRegistration(RegistrationManager.java:128)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.registerService(AbstractComponentManager.java:959)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.activateInternal(AbstractComponentManager.java:732)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.enableInternal(AbstractComponentManager.java:666)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.AbstractComponentManager.enable(AbstractComponentManager.java:432)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.manager.ConfigurableComponentHolder.enableComponents(ConfigurableComponentHolder.java:665)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.BundleComponentActivator.initialEnable(BundleComponentActivator.java:338)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator.loadComponents(Activator.java:382)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator.access$200(Activator.java:49)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator$ScrExtension.start(Activator.java:264)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.createExtension(AbstractExtender.java:196)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.modifiedBundle(AbstractExtender.java:169)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.addingBundle(AbstractExtender.java:139)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.addingBundle(AbstractExtender.java:49)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.osgi.util.tracker.BundleTracker$Tracked.customizerAdding(BundleTracker.java:475)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.osgi.util.tracker.BundleTracker$Tracked.customizerAdding(BundleTracker.java:420)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.osgi.util.tracker.AbstractTracked.trackAdding(AbstractTracked.java:256)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.osgi.util.tracker.AbstractTracked.trackInitial(AbstractTracked.java:183)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.osgi.util.tracker.BundleTracker.open(BundleTracker.java:159)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.startTracking(AbstractExtender.java:100)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.doStart(AbstractExtender.java:92)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator.doStart(Activator.java:172)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.AbstractExtender.start(AbstractExtender.java:72)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator.restart(Activator.java:139)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.config.ScrConfigurationImpl.configure(ScrConfigurationImpl.java:217)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.config.ScrConfigurationImpl.start(ScrConfigurationImpl.java:120)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.apache.felix.scr.impl.Activator.start(Activator.java:100)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:815)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl$3.run(BundleContextImpl.java:808)
INFO | jvm 1 | 2022/05/20 02:38:40 | at java.security.AccessController.doPrivileged(Native Method)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.startActivator(BundleContextImpl.java:808)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.BundleContextImpl.start(BundleContextImpl.java:765)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.EquinoxBundle.startWorker0(EquinoxBundle.java:1005)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.internal.framework.EquinoxBundle$EquinoxModule.startWorker(EquinoxBundle.java:357)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.Module.doStart(Module.java:589)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.Module.start(Module.java:457)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1784)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.incStartLevel(ModuleContainer.java:1764)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.doContainerStartLevel(ModuleContainer.java:1726)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1657)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.container.ModuleContainer$ContainerStartLevel.dispatchEvent(ModuleContainer.java:1558)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventManager.java:233)
INFO | jvm 1 | 2022/05/20 02:38:40 | at org.eclipse.osgi.framework.eventmgr.EventManager$EventThread.run(EventManager.java:343)
INFO | jvm 1 | 2022/05/20 02:39:00 | [Fri May 20 02:39:00.675 PDT 2022] [webMethods Integration Server] Setting log level to ERROR
INFO | jvm 1 | 2022/05/20 02:39:00 | --------- Log File Closed ---------

4

I can see only this one as suspicious log from wrapper.log … remaining all logs are relevant to the insert SQL Queries.

5

Also i tried from the different browsers , IS is not reachable

6

You’ve mentioned that the IS process is running, is restarting an option for you?

Do you see this message in the server logs?
And when you try to reach IS , are there any messages being generated in the server logs?
-NP

7

yes i can see this message Enabling HTTP Listener on port 8662 … But here 8662 is not our primary port number.

8

Our Primary port is 8668 , So in the logs it should be right …
But in logs i dont see this port … only i can see for 8662
2022-05-20 02:41:58 PDT [TNS.0005.1042I] Warning: Trying to register delivery service FTP AMEX to BOA that is already registered
2022-05-20 02:41:58 PDT [ISS.0028.0012I] AppleTNInfrastructure: Startup service (com.apple.b2b.tninfrastructure.sap:registerRouteOnly)
2022-05-20 02:41:58 PDT [ISS.0028.0013E] Startup service exited with the following error: java.lang.NullPointerException
2022-05-20 02:41:58 PDT [ISS.0028.0012I] AppleTNInfrastructure: Startup service (com.apple.b2b.tninfrastructure.webService.soapProcessor:registerCustomSOAPmsgProcessor)
2022-05-20 02:41:58 PDT [ISP.0046.0012I] Enabling HTTP Listener on port 8662
2022-05-20 02:41:58 PDT [ISS.0098.0021I] Persistent Trigger Output Dispatcher started
2022-05-20 02:41:58 PDT [ISS.0098.0021I] Volatile Trigger Output Dispatcher started
2022-05-20 02:41:58 PDT [ISS.0098.0027I] PersistenceManager started all Stores
2022-05-20 02:41:58 PDT [ISS.0025.0036I] Dispatcher started
2022-05-20 02:41:58 PDT [ISS.0025.0013I] Cache Sweeper started
2022-05-20 02:41:58 PDT [ISS.0035.0003I] Log rotation interval set to 1440 minutes.
2022-05-20 02:41:58 PDT [ISS.0014.0025W] Master password for outbound password encryption has expired. Password was last changed on 2020-03-27 11:21:01 PDT. The current expiration interval is 90 days
2022-05-20 02:41:58 PDT [ISS.0025.0057I] Reading system level information
2022-05-20 02:41:58 PDT [ISS.0014.0002I] Initialization completed in 192 seconds.
2022-05-20 02:41:58 PDT [SAP.0118.0001W] Adapter is not using Centralized Transaction Store.
2022-05-20 02:41:58 PDT [ISS.0025.0016I] Config File Directory Saved

9

You can connect to the port that is enabled and check the Security->Ports screen to check for the primary port that doesn’t seem to be enabled.
-NP

1 Like
10

Resolution: listeners.cnf file is corrupted, ports are changed. We have corrected that file and restarted the server.
It works

1 Like
11

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.