MWS Not Getting Started - wM 10.5

Dear Expert,
I am getting below error while starting MWS. Things were working fine until one day where suddenly it went down with below error(invalid credential for “webm_clusteruser”). Appreciate any help here -

2021-06-18 09:30:25 PDT (c.w.p.s.i.PostInitStatus:INFO) [MWS Startup-1] - My webMethods Server “default” Node “********node10421” took 63 seconds to initialize
role: notification
role: search
role: taskengine
role: autodeploy
http listening at: hostname:8585
FrontEndUrl: hostname:8585

2021-06-18 09:30:25 PDT (c.w.p.s.i.i.DefaultPhase:INFO) [MWS Startup-1] - Initializing component: com.webmethods.portal.bizPolicy.biz.install.impl.InstallFixes
2021-06-18 09:30:25 PDT (c.w.p.s.i.i.DefaultPhase:INFO) [MWS Startup-1] - Initializing component: com.webmethods.jackrabbit.spi2mws.SearchIndexInitializer
2021-06-18 09:30:25 PDT (c.w.j.s.SearchIndexInitializer:INFO) [MWS Startup-1] - Initializing the search index asynchronously
2021-06-18 09:30:26 PDT (c.w.p.s.i.i.DefaultPhase:INFO) [MWS Startup-1] - Initializing component: com.webmethods.portal.system.impl.CleanupTempFiles
2021-06-18 09:30:26 PDT (c.w.p.s.i.CleanupTempFiles:INFO) [Thread-62] - Cleaning up temp folder …
2021-06-18 09:30:26 PDT (c.w.p.p.i.w.CurrentSessionsListener:INFO) [MWS Startup-1] - Activating service
2021-06-18 09:30:26 PDT (c.w.p.p.i.w.GetContextViewList:INFO) [MWS Startup-1] - Activating service
2021-06-18 09:30:26 PDT (c.s.w.o.a.s.i.TransportListenerRegistry:INFO) [MWS Startup-1] - WSSOGI0028: Registering a TransportListener for connector {com.webmethods.portal.system.cluster.IServer}={port=8585, scheme=http, service.id=1596, service.bundleid=210, service.scope=singleton}
2021-06-18 09:30:26 PDT (E.S.m.p.wm_sessionmonitor:INFO) [MWS Startup-1] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (E.S.m.p.wm_sessionmonitor:INFO) [MWS Startup-1] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (E.S.c.s.w.o.a.setup:INFO) [MWS Startup-1] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (E.S.c.w.c.server:INFO) [MWS Startup-1] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (E.S.c.s.w.osgi:INFO) [pool-7-thread-2] - ServiceEvent UNREGISTERING
2021-06-18 09:30:26 PDT (c.s.w.o.a.e.i.AxisModuleBundleExtension:INFO) [pool-7-thread-2] - WSSOGI0017: Stopped module wssstat
2021-06-18 09:30:26 PDT (E.S.c.s.w.o.axis2:INFO) [pool-7-thread-2] - ServiceEvent UNREGISTERING
2021-06-18 09:30:26 PDT (c.w.p.w.w.PortalArtifactFactory:INFO) [MWS Startup-1] - MWS is not configured to use third party IDPs for SSO
2021-06-18 09:30:26 PDT (c.s.w.o.a.i.OsgiAxisConfigurator:INFO) [pool-7-thread-2] - WSSOGI0011: Using Axis2 configuration file file:/app/softwareag/MWS/server/default/config/wss/axis2.xml
2021-06-18 09:30:26 PDT (c.s.w.o.a.i.OsgiAxisConfigurator:INFO) [pool-7-thread-2] - WSSOGI0012: Using Axis2 repository /app/softwareag/profiles/MWS_default/workspace/wsstack/repository
2021-06-18 09:30:26 PDT (E.S.c.s.w.osgi:INFO) [pool-7-thread-2] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (c.s.w.o.a.e.i.AxisModuleBundleExtension:INFO) [pool-7-thread-2] - WSSOGI0014: Created module wssstat in bundle com.softwareag.web.caf.mws.wsstat.mar_10.5.0.0000-0232 [177]
2021-06-18 09:30:26 PDT (c.s.w.o.a.e.i.AxisModuleBundleExtension:INFO) [pool-7-thread-2] - WSSOGI0015: Starting any discovered modules
2021-06-18 09:30:26 PDT (c.w.p.w.w.WSStackClientInitializer$WssServiceListener:INFO) [pool-7-thread-2] - MWS WS-Stack client init: found a registered Axis2 configuration context
2021-06-18 09:30:26 PDT (E.S.c.s.w.o.axis2:INFO) [pool-7-thread-2] - ServiceEvent REGISTERED
2021-06-18 09:30:26 PDT (c.w.j.s.q.l.SearchIndex:INFO) [pool-12-thread-1] - Index initialized: /app/softwareag/MWS/server/default/data/search_jcr Version: 3
2021-06-18 09:30:29 PDT (c.w.p.s.d.g.i.GdprConfigService:INFO) [Timer-9] - Added GDPR IDistributedConfigChangedEvent listener
2021-06-18 09:30:29 PDT (c.w.p.s.d.g.i.UserPrivacyService:INFO) [Timer-9] - Successfully added GDPR subscribers
2021-06-18 09:30:37 PDT (c.w.p.f.i.PortalServlet:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Request [1b8u4ts0agtn7apxnvca5yvvi:null] http://localhost:8585/ (POST)
2021-06-18 09:30:37 PDT (c.w.p.f.s.h.Validate:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Trying to authenticate user: WEBM_CLUSTERUSER
2021-06-18 09:30:37 PDT (c.w.p.p.w.s.SystemDirPrincipalProvider:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Authenticating: webm_clusteruser
2021-06-18 09:30:37 PDT (c.w.p.m.d.i.DirSystemMechanics:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Authentication failed for user WEBM_CLUSTERUSER: [POP.003.0141] The username/password you entered is invalid.
2021-06-18 09:30:38 PDT (c.w.p.f.s.h.Validate:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Validate::handle() - failed to login
2021-06-18 09:30:38 PDT (c.w.p.f.s.h.Validate:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Trying to authenticate user: WEBM_CLUSTERUSER
2021-06-18 09:30:38 PDT (c.w.p.p.w.s.SystemDirPrincipalProvider:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Authenticating: webm_clusteruser
2021-06-18 09:30:38 PDT (c.w.p.m.d.i.DirSystemMechanics:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Authentication failed for user WEBM_CLUSTERUSER: [POP.003.0141] The username/password you entered is invalid.
2021-06-18 09:30:38 PDT (c.w.p.f.s.h.Validate:INFO) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Validate::handle() - failed to login
2021-06-18 09:30:38 PDT (c.w.r.u.h.ParametersUtil:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] - RequestURI: /
2021-06-18 09:30:38 PDT (c.w.r.u.h.ParametersUtil:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] - RequestPath: /
2021-06-18 09:30:38 PDT (c.w.r.u.h.ParametersUtil:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] - Parameters: {axsrft=bosq4rhl0k4s2bykp7h673sqf, remoteContextID=/meta/default/user/0000000001, base64EncodedCommandBean=base64EncodedCommandBean, base64EncodedKey=base64EncodedCommandBean, serializeResult=true, command=ping}
2021-06-18 09:30:38 PDT (c.w.r.u.h.ParametersUtil:WARN) [qtp16959078-90][ [RID:1]] [ [RID:1]] -

Hi Kumar,

just for clarification:
Did you change the password for webm_clusteruser or was its login availability disabled by some meanings?
We never touched these webm* user in any way and were not observing such messages.

Regards,
Holger

Hi Holger,
Yeah that’s surprising for me as we never change any credential for webm_clusteruser . Is there anyway to replace any config file to fix this issue ? We can get another config file from another server already up and running in different envrionment.

Hi Kumar,

I am not sure if just replacing a config file will help you out here.

Might be a good idea to open a support incident in Empower to get this checked by Support Team.

One thing I can remember might be that the user got somehow disabled for login in the users settings.
Hopefully this can be solved with an update to MWS database schema when finding the right column and table where this flag is stored.

Final solution (when nothing helps and hopefully there is no live data in the MWS database schema, except for its own config):
Uninstall this MWS instance, recreate the database schema and reinstall and reconfigure MWS afterwards.

Regards,
Holger

Thanks everyone for your response. Ultimately we tried restarting the linux server and this issue got resolved. Looks like there is changes done on linux server that may have caused this issue.
Appreciate responses from all of you.