MWS logs are frequently increasing

Hello Experts,

We are using 10.5 version, our MWS logs are frequently getting increased and our disk space is completely get occupied and server is going down. Could you please someone assist to overcome from this issue.

Example:
18G full.log
18G problems.log

Thanks

Give the documentation a check, you can adjust the logging level if it’s set to high in your environment. However, if it’s already at an acceptable level, then you’ll have to investigate and resolve the problems.

Can you post your top 3 or 5 most frequent errors?

KM

HI Kasi,

Thank you for response :slight_smile: , please refer the frequent errors from full log.

on_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
at com.webmethods.rtl.event.QueueDeliverer$EventQueue.processEvent(QueueDeliverer.java:381) ~[com.webmethods.caf.shared.event.common_10.5.0.0009-0152.jar:10.5.0.0009-0152]
2021-10-13 17:01:43 CEST (c.w.p.j.d.i.JMSTopicConnection:ERROR) [MWS JMSConnection pool-12-thread-1] - [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-01502: index ‘WM9_EE_SIT_MWS.PK_T_JMS_EVENTS’ or partition of such index is in unusable state

java.sql.SQLException: [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-01502: index ‘WM9_EE_SIT_MWS.PK_T_JMS_EVENTS’ or partition of such index is in unusable state

    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.b(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oracle.ddam.u(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.dder.y(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.dder._(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.dddr.executeUpdate(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.webmethods.portal.service.sql.core.BaseSqlWorker.executeUpdate(BaseSqlWorker.java:324) ~[com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.service.sql.core.BaseSqlWorker.executeUpdate(BaseSqlWorker.java:374) ~[com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.service.sql.core.BaseSqlWorker.executeUpdate(BaseSqlWorker.java:283) ~[com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.jms.db.impl.oracle.EventSave.executeQuery(EventSave.java:30) ~[com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.service.sql.core.BaseSqlWorker.executeQueryProcOut(BaseSqlWorker.java:809) ~[com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.jms.db.impl.JMSConnection.drainTopicQueue(JMSConnection.java:343) [com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.jms.db.impl.JMSConnection$TopicQueueTimerTask.call(JMSConnection.java:246) [com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at com.webmethods.portal.jms.db.impl.JMSConnection$TopicQueueTimerTask.call(JMSConnection.java:243) [com.webmethods.caf.server_10.5.0.0009-0152.jar:10.5.0.0009-0152]
    at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

2021-10-13 17:01:43 CEST (c.w.p.j.d.i.o.RegisterSubscription:INFO) [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1] - Found existing subscription: 805 for .com.webmethods.portal.notification.portal.msg.queue.messages
2021-10-13 17:01:43 CEST (c.w.p.s.c.i.Server:INFO) [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1] - Updated Maintenance Mode (serverName: 10.156.150.165-node9947):
maintenanceReason → from JMS system is down to null
isMaintenanceMode → from true to false

2021-10-13 17:01:43 CEST (e.LogSupport:WARN) [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1] - ServiceTime.start(), bad state, caller has program logic error, fix it! [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1]
2021-10-13 17:01:44 CEST (c.w.p.j.d.i.JMSQueueConnection:WARN) [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1] - Failed to publish message: com.webmethods.portal.service.meta2.system.impl.ScheduleEvent
2021-10-13 17:01:44 CEST (c.w.p.j.d.i.JMSQueueConnection:WARN) [MWS Event Queue-com.webmethods.portal.notification.portal.msg.queue.messages (Outbound pool-16-thread-1] - [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-01502: index ‘WM9_EE_SIT_MWS.PK_T_JMS_EVENTS’ or partition of such index is in unusable state

java.sql.SQLException: [SoftwareAG][Oracle JDBC Driver][Oracle]ORA-01502: index ‘WM9_EE_SIT_MWS.PK_T_JMS_EVENTS’ or partition of such index is in unusable state

    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.b(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.ddcj.a(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oracle.ddam.u(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]
    at com.wm.dd.jdbc.oraclebase.dder.y(Unknown Source) ~[com.softwareag.ext.dd.cjdbc_6.0.0.0002-0002.jar:?]

Hi Varun,

please check with Oracle SQL Developer or any other database client which can handle Oracle Databases why the index for the mentioned primary key is in an unusable state.
There might the need to get it refreshed.
If you are not sure what is the reason you can get in contact with your DBA and ask them for assistance.

If nothing helps you might have to recreate the database schema for MWS which in turn means that it might be neccessary to recreate and reconfigure the complete MWS instance. This can result in data loss when this instance is meant for TaskEngine usage.

Regards,
Holger

1 Like

Hi @Holger_von_Thomsen ,

Thank you for response :slight_smile: , Sure, i will check with the DB team, In BW is it possible to clear the MWS logs ?

Thank you
KMV

Hi Varun,

yes, this will be possible.

Shutdown the affected MWS instance.
Once finished you can move/archive the logs somewhere else and then restart the instance.
Additionally check for the profiles/MWS_default/logs directory if there are other large log files.

Regards,
Holger

HI @Holger_von_Thomsen ,

Thank you so much for guidance :slight_smile: