Can't startup webMethods Integration Server

abaabbad · January 2, 2022, 10:01am

What product/components do you use and which version/fix level?

Integration Server v10.5

Are you using a free trial or a product with a customer license?

No, I’m using partner product.

What are trying to achieve? Please describe in detail.

Start Integration Server

Do you get any error messages? Please provide a full error message screenshot and log file.

2022-01-02 11:24:32 AST [ISS.0025.0001I] Integration Server 10.5.0.0 Build 84 
2022-01-02 11:24:32 AST [ISS.0025.0006I] License Manager started 
2022-01-02 11:24:32 AST [ISS.0001.0008W] Microservices Functionality disabled due to license restrictions. 
2022-01-02 11:24:32 AST [ISS.0025.0049I] The JCE Unlimited Strength Jurisdiction Policy File was found 
2022-01-02 11:24:32 AST [ISS.0025.0041I] FIPS mode not initialized 
2022-01-02 11:24:33 AST [ISS.0025.0024I] JDBC Connection Manager started 
2022-01-02 11:24:34 AST [ISS.0095.0013I] Audit Logging initialized. 
2022-01-02 11:24:41 AST [ISS.0033.0168I] Cluster Node Name: DESKTOP-9K1R8FQ. 
2022-01-02 11:24:41 AST [BAS.0123.0001I] FSData full consistency check is being performed. Db:WmRepository4\RepoV4 
2022-01-02 11:24:42 AST [ISS.0025.0017I] Repository Manager started 
2022-01-02 11:24:42 AST [ISS.0025.0021I] ACL Manager started 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Users" to ACL "Anonymous" because group "My webMethods Users" is not found. 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Users" to ACL "Default" because group "My webMethods Users" is not found. 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Administrators" to ACL "Administrators" because group "My webMethods Administrators" is not found. 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Administrators" to ACL "Internal" because group "My webMethods Administrators" is not found. 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Administrators" to ACL "Developers" because group "My webMethods Administrators" is not found. 
2022-01-02 11:24:42 AST [ISS.0024.0015E] Integration Server cannot add group "My webMethods Administrators" to ACL "Replicators" because group "My webMethods Administrators" is not found. 
2022-01-02 11:24:42 AST [ISS.0025.0008I] State Manager started 
2022-01-02 11:24:42 AST [ISS.0025.0010I] Service Manager started 
2022-01-02 11:24:42 AST [ISS.0025.0020I] Validation Processor started 
2022-01-02 11:24:42 AST [ISS.0025.0022I] Statistics Processor started 
2022-01-02 11:24:42 AST [ISS.0025.0018I] Invoke Manager started 
2022-01-02 11:24:42 AST [ISS.0025.0012I] Cache Manager started 
2022-01-02 11:24:42 AST [BAS.0123.0001I] FSData full consistency check is being performed. Db:DocumentStore\ISTransStore 
2022-01-02 11:24:42 AST [ISS.0098.0026I] Document Store DefaultStore initialized 
2022-01-02 11:24:43 AST [BAS.0123.0001I] FSData full consistency check is being performed. Db:DocumentStore\TriggerStore 
2022-01-02 11:24:44 AST [ISS.0098.0026I] Document Store VolatileTriggerStore initialized 
2022-01-02 11:24:44 AST [ISS.0098.0026I] Document Store TriggerStore initialized 
2022-01-02 11:24:44 AST [BAS.0123.0001I] FSData full consistency check is being performed. Db:DocumentStore\ISResubmitStore 
2022-01-02 11:24:44 AST [ISS.0098.0026I] Document Store ResubmitStore initialized 
2022-01-02 11:24:45 AST [ISS.0153.0007I] webMethods Messaging connection alias IS_UM_CONNECTION started 
2022-01-02 11:24:45 AST [ISS.0153.0007I] webMethods Messaging connection alias IS_DES_CONNECTION started 
2022-01-02 11:24:45 AST [ISS.0153.0000I] webMethods Messaging subsystem started 
2022-01-02 11:24:45 AST [ISS.0106.0001I] Join Manager initialized 
2022-01-02 11:24:45 AST [ISS.0025.0032I] Dispatcher initialized 
2022-01-02 11:24:45 AST [ISS.0134.0128I] JMS subsystem is starting. 
2022-01-02 11:24:45 AST [ISS.0134.0998I] DEFAULT_IS_JMS_CONNECTION cache enabled :: true 
2022-01-02 11:24:45 AST [ISS.0134.0998I] PE_NONTRANSACTIONAL_ALIAS cache enabled :: true 
2022-01-02 11:24:45 AST [ISS.0134.0000I] JMS subsystem started 
2022-01-02 11:24:45 AST [ISS.0173.0000I] Common Messaging subsystem started 
2022-01-02 11:24:45 AST [BAS.0123.0001I] FSData full consistency check is being performed. Db:XAStore\DefaultXAStore 
2022-01-02 11:24:45 AST [SCC.0121.0093I] XA Recovery Store initialized 
2022-01-02 11:24:45 AST [com.webmethods.fabric.agent.FabricAgent.null.load.startI] [agent.load.start] Loading registry config file C:\SoftwareAG\IntegrationServer\instances\default\config\fabric-agent.xml 
2022-01-02 11:24:45 AST [com.webmethods.fabric.agent.FabricAgent.null.startI] [agent.start] Starting Fabric Agent 
2022-01-02 11:24:45 AST [ISS.0025.0055I] Lock Manager initialized 
2022-01-02 11:24:46 AST [ISS.0014.0054C] Error occurred while initializing server: java.lang.NoSuchMethodError: org.apache.logging.log4j.spi.LoggerContextFactory.isClassLoaderDependent()Z	at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:49)	at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46)	at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33)	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358)	at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383)	at org.opensaml.DefaultBootstrap.getLogger(DefaultBootstrap.java:246)	at org.opensaml.DefaultBootstrap.initializeXMLSecurity(DefaultBootstrap.java:189)	at org.apache.rahas.RampartSAMLBootstrap.bootstrap(RampartSAMLBootstrap.java:76)	at org.apache.rahas.Rahas.init(Rahas.java:41)	at org.apache.axis2.context.ConfigurationContextFactory.initModules(ConfigurationContextFactory.java:252)	at org.apache.axis2.context.ConfigurationContextFactory.init(ConfigurationContextFactory.java:230)	at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContext(ConfigurationContextFactory.java:93)	at org.apache.axis2.context.ConfigurationContextFactory.createConfigurationContextFromFileSystem(ConfigurationContextFactory.java:210)	at com.wm.app.b2b.server.Server.initWSStack(Server.java:1080)	at com.wm.app.b2b.server.Server.run(Server.java:653)

wolfgang.winter · January 2, 2022, 11:37am

Did you replace log4j components within Integration Server on your own ?

abaabbad · January 2, 2022, 12:03pm

No, I just update the fix through update manager.

abaabbad · January 2, 2022, 2:26pm

It seems the issue coming from Shared Libraries Loggers 10.5 Fix 2 10.5.0.0002-0247

Nagendra_Prasad · January 2, 2022, 3:48pm

If the issue can clearly be attributed to a fix as you have found out , it is best to raise a support ticket mentioning details of fixes applied.
-NP

Bharath_V.K · January 3, 2022, 8:04am

Hi @abaabbad,

Can you please check if OSGI (Platform) Fix 18 is installed? If it is not, please try installing this fix and check if IS starts.

Thank you.

abaabbad · January 3, 2022, 8:42am

Hi @Bharath_V.K,

I was update all fixes to the latest version.

Holger_von_Thomsen · January 3, 2022, 4:50pm

Please keep in mind that there is currently a severe vulnerabilitiy in log4j 2 API.

There is an article in Empower describing the affected versions and possibility how to mitigate.

Regards,
Holger

Veaceslav_Samohvalov · February 14, 2022, 11:51am

Good day.
We have exactly the same error message using the Integration Server 10.11.0.0 Build 724 both in Linux and in Windows environment.
How to reproduce:

Install Designer as usual, e.g. using SoftwareAGInstaller20220121-w64.exe from Empower.
Verify that the “default” instance of IS is running properly
Create new instance of IS by this command:
./is_instance.bat create -Dinstance.name=test
For simplicity I omitted other options and added the license (unlimited) manually.
Shutdown all running instances of IS
Start new IS:
/c/SoftwareAG_10_11/profiles/IS_test/bin/startup.bat

You get this error (full server.log attached) - IS crashed:
2022-02-14 12:34:00 CET [ISS.0014.0054C] (tid=93) Error occurred while initializing server: java.lang.NoSuchMethodError: ‘boolean org.apache.logging.log4j.spi.LoggerContextFactory.isClassLoaderDependent()’ at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:49) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:48) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at com.maverick.ssh.SshConnector.(SshConnector.java:142) at com.maverick.ssh.LicenseManager.addLicense(LicenseManager.java:18) at com.wm.app.b2b.server.sftp.client.SFTPClientManager.initLicense(SFTPClientManager.java:118) at com.wm.app.b2b.server.sftp.client.SFTPClientManager.init(SFTPClientManager.java:138) at com.wm.app.b2b.server.sftp.client.SFTPClientManager.initialize(SFTPClientManager.java:110) at com.wm.app.b2b.server.Server.run(Server.java:655)

We would appreciate if somebody could give some hints on what is going on here.
Thank you.

server.log (2.7 KB)

Venkata_Kasi_Viswanath_Mugada1 · February 14, 2022, 12:48pm

I recommend creating a support ticket for this, particularly since you’ve identified the repro steps already.
Once you resolve it, do post the root cause and resolution to help others.

KM

Kalpesh_Shah1 · February 14, 2022, 1:55pm

Can you please ensure that you reinstall the required platform fixes in your environment? This problem exists particularly when you create new instance after applying the fixes - webMethods Fixes for Log4j Zero-Day security vulnerability - Service project (softwareag.com) - Please make sure that following fixes are reinstalled.

Third-Party Libraries > Shared Libraries Loggers 10.11 Fix 2
Shared Bundles > Shared Bundles Loggers 10.11 Fix 2
Infrastructure > Shared Platform > Platform Diagnostic 10.11 Fix 1

Regards,
-Kalpesh

Veaceslav_Samohvalov · February 14, 2022, 5:15pm

Thanks Kalpesh. You nailed it. We did our Designer installation on 27.01.2022. The latest fixes came out on 09.02.2022.

Trial 1:

Install all latest fixes using SAGUpdateManager.
Start new IS (which was not working): /c/SoftwareAG_10_11/profiles/IS_test/bin/startup.bat
=> SUCCESS!
Start Designer … and oh no …

!ENTRY org.eclipse.core.jobs 4 2 2022-02-14 15:35:25.443
!MESSAGE An internal error occurred during: “Initializing Java Tooling”.
!STACK 0
java.lang.VerifyError: Bad type on operand stack
Exception Details:
Location:
org/eclipse/jdt/internal/core/search/matching/MatchLocator.reportMatching(Lorg/eclipse/jdt/internal/compiler/ast/FieldDeclaration;[Lorg/eclipse/jdt/internal/compiler/ast/FieldDeclaration;Lorg/eclipse/jdt/internal/compiler/ast/TypeDeclaration;Lorg/eclipse/jdt/core/IJavaElement;IZLorg/eclipse/jdt/internal/core/search/matching/MatchingNodeSet;)V @343: invokevirtual
Reason:
Type ‘java/lang/Object’ (current frame, stack[1]) is not assignable to ‘org/eclipse/jdt/internal/compiler/ast/ASTNode’

Trial 2:
0. Uninstall ALL fixes using SAGUpdateManager.

Install all latest fixes using SAGUpdateManager.
Start new IS (which was not working): /c/SoftwareAG_10_11/profiles/IS_test/bin/startup.bat
=> SUCCESS!
Start Designer
=> SUCCESS!

We plan to do the same on our Linux server. I will report once we have good news.

Thank you very much again for your quick support.

Best regards,
Veaceslav

Veaceslav_Samohvalov · February 14, 2022, 8:19pm

Well. I discovered a strange behaviour here. The instance “test”, which was not working before the update, began to work after the update. And it is still working. But …
When I created afterwards a brand new instance “uat”, I got exactly the same error as described above.
I could reproduce the same behavior in our linux environment.
Bottomline: the update could heal corrupt instances, but cannot do anything for all new instances.
Any ideas?

Kalpesh_Shah1 · February 14, 2022, 9:21pm

@Veaceslav_Samohvalov , as I mentioned earlier, it is required to reinstall the fixes to ensure OSGi platform fixes are available in newly created instance. Did you try reinstalling the fixes mentioned earlier in newly created instance?

Veaceslav_Samohvalov · February 15, 2022, 10:59am

@Kalpesh_Shah1, thank you for clarification. I misinterpreted your first comment. So, the correct interpretation would be, as you mentioned already, to reinstall the fixes mentioned earlier in ALL newly created instances.
I can confirm, after the reinstallation of the fixes the new “corrupted” “uat”-instance started to load w/o problems.

Thanks again for your quick support.

Jesus_Carrillo1 · February 25, 2022, 3:58am

Hi, I have the same problem. How did u solve it?
In mi case I have IS 10.5 Version
I’ve installed IS_10.5_Core_Fix13 then IS can’t start. Ive finded this thread and ive installed:
Third-Party Libraries > Shared Libraries Loggers 10.11 Fix 2
S~~hared Bundles > Shared Bundles Loggers 10.11 Fix 2~~ I COULDN’T FIND THIS ONE
Infrastructure > Shared Platform > Platform Diagnostic 10.11 Fix 1

My instance is the default one. My error is:

Error occurred while initializing server: java.lang.NoSuchMethodError: org.apache.logging.log4j.spi.LoggerContextFactory.isClassLoaderDependent()Z at org.apache.logging.slf4j.Log4jLoggerFactory.getContext(Log4jLoggerFactory.java:49) at org.apache.logging.log4j.spi.AbstractLoggerAdapter.getLogger(AbstractLoggerAdapter.java:46) at org.apache.logging.slf4j.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:33) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:383) at com.maverick.ssh2.Ssh2Context.(Ssh2Context.java:58) at com.wm.app.b2b.server.sftp.client.SFTPClientCommonConfig.setMacC2S(SFTPClientCommonConfig.java:93) at com.wm.app.b2b.server.sftp.client.SFTPClientCommonConfig.loadDefaultConfiguration(SFTPClientCommonConfig.java:34) at com.wm.app.b2b.server.sftp.client.common.SFTPClientSelfMigrator.migrate(SFTPClientSelfMigrator.java:66) at com.wm.app.b2b.server.sftp.client.common.SFTPClientSelfMigrator.migrate(SFTPClientSelfMigrator.java:212) at com.wm.app.b2b.server.Server.run(Server.java:641)

Thanks

Holger_von_Thomsen · February 25, 2022, 11:19am

Hi Jesus,

remember to (re-)apply the TPP_10.5_Misc_Fix3 as well to use the updated JSch lib which is used by the SFTP feature.

Regards,
Holger

Veaceslav_Samohvalov · February 25, 2022, 11:56am

@Jesus_Carrillo1 :
Hi Jesus,
As mentioned already by Kalpesh and Thomas above, you have to reinstall the fixes mentioned earlier. Probably for the version 10.5 one has another set of fixes. I reinstalled the fixes using SUM with the “-overInstall true” flag.
Hope this helps.

Regards,
Veaceslav

Jesus_Carrillo1 · February 25, 2022, 5:38pm

Hi, Veaceslav and Holger.
I’ve overinstalled that fixes and server.

But when I’ve started the IS it failed again with the same error.

Veaceslav_Samohvalov · February 25, 2022, 5:49pm

Hi Jesus,

Since we use the version 10.11 I cannot give you any hints on 10.5. But today I had some problems with the fixes too (it did not work at first). So I carried out a full update with all latests fixes. Then reinstalled the fixes related to log4j and it worked out. So maybe you can give it a try (if not done already). Otherwise, somebody from the SAG team should comment.

Best regards.