Process instances no longer show on MWS after triggering Process Archive

Hello everybody,

I faced the following problem:
There is no more process instances on MWS after the audit log archive fails. Processes are still created and can be seen in the database, but as of that date, 12/29/2023, all the new processes are no longer visible on the process instance tab in MWS. Also, service execution is no longer displayed in MWS - services tab or on the IS monitoring page. I also mentioned that WMSERVICE_MIN_MAX table stopped filling after that date, but WMSERVICE and WMSERVICECUSTOMFLDS still work.

I didn’t notice a specific error in the IS/MWS logs, but when a task is found through task management for its process, the data looks like this:
Process:
Start Date / Time:01/01/1970 01:00:00.000
Last Updated:01/01/1970 01:00:00.000
Instance ID:
Instance Iteration:1
Status:Broker (deprecated)
Duration:0d 00:00:00.000

I am currently using webMethods version 10.5 of IS/MWS components under a user license, which is already updated with the latest fixes (software/database). During update, the database components were migrated, nothing was recreated.

Could you please suggest what else I should try ?

Kind regards,
Milan

I would suggest contacting SAG support for this.

Hi Milan

did you check for the tables beginning with “PRA_*”, which are meant to contain the process audit data?

How did you notice that the audit log archive failed?
In WmMonitor there are several services for archiving/deleting data from the audit log db schema.

See WmMonitor Users Guide for details.

Regards,
Holger

Did you run archive as Stored procedure or as a service? If you executed it as an SP it might still be working in the DB.

This might be incorrect, try filtering the search criteria so that less results will be returned and check again. Whether you archive your data or not, MWS should be able to bring that data. Only relation with archive and process instances is it will take a lot longer to bring process results if you don’t archive frequently, hence this might be the case here. For instance, try searching with a specific process ID or limit it to half an hour or so. It should return results.

If you executed archive as SP (there is a parameter in the service which sets it to run as SP or as service) it can still be running if you haven’t archived your DB for a very long time. Did you check operations table? If you check archive DB you can see the operations table, and current status of the procedure (if its still running or not, or what happened when you executed it last time). You can update that value to stop SP execution and trigger it again. It will clear the status of SP.

What you explained so far made me believe the SP is still running. For a better comparison I had a db with 8TB non archived data. My archive SP took more then a week (I don’t remember how long exactly it took but it was between a month and a week), When we executed it first time, it really looked like it was stuck and we canceled the SP execution by updating the operation_control table. It turned out that we didn’t really need to do it though. we could forget about it and it would be finished. Instead, we limited the SP archive window programmatically and run it incrementally (as archive 01/01/2024, and then 02/01/2024 and so on). When we had that bloated DB process tables, any queries were taking several minutes to finish executing. If you executed it as service and if you canceled it it might be stopped but it should start running next time you trigger the same service again. Its usually preferred to archive using DB only, if you triggered this it will keep running until the first query finishes executing. Consequent calls will only update the parameters of the initial SP query. I am not sure if it changes the execution of the current ongoing SP execution though.

The table below controls the SP execution. The operation_status column will have the current status of the last execution. It will tell you if its running or failed etc. If its complete, that means your arhive SP has finished work.
**You can change operation_status to COMPLETE to stop execution, but do it only as last resort! You need to executre it again if you stop it this way. **

Table below shows last operation. You should be seeing lots of entries here if it is successfully running. You should see this table updating if the SP is running.

You can find the last execution parameters from the table below if you need them

You can also monitor the size of the DBs to verify if its running or not. These screenshots are taken from Sql Server, Oracle has a similar structure, Other DBs might be different. Also keep in mind that after archive, you need a shrink especially if its the first time archiving after a very long time. If you don’t shrink the DB your performance will be effected. Here in your case, it looks like your archive SP is still running and its degrading your performance, hence its taking longer to finish.

Thanks for the quick and detailed replies.

I tried to archive through the GUI in the IS monitor section, for the first time after half a year of use. It worked for a while and then it threw out some logs. I attached the logs appeared in the “monitoring_logs” file.

It doesn’t seem like the archiving is still going on, because according to these logs and the review of the database according to your instructions, seems it was started and broke at a certain moment. I’m not sure if anything was archived. But I noticed these two tables stopped filling up with data from that point: PRA_PROCESS_RECENT & PRA_PROCESS_ACTION.

I am attached most of the relevant tables in the hope of spotting something specific:
monitoring_logs.txt (2.1 KB)

operation_log_parameter.zip (4.3 KB)

Thank you!

It triggers the same service in the background. It doesn’t matter where you trigger archive. Only important parameter when triggering archive process is whether run it on DB or IS. If you set it to run in DB then even shutting down the IS won’t terminate it. Since you said the operation_status is complete in operatin_control table, it should have been terminated already.

If the archive process is terminated and the tables are populating and even limiting the query won’t return any results, that probably means there is a some kind of a synchronization issue. Please verify if you have patched the DB as well for the fixes you have installed, check the DB connections for MWS and IS, WmMonitor settings and make sure you have a different DB setup for archive (unless you set it to purge instead of archive). If none of these help, you may need to create a ticket but you will need an extended support agreement for version 10.5 since it is out of support since October.

Hi,

a second DB is not really required, but a separate schema for the Archiving function, as real archiving will move the data from the main schema to the archive schema. Even when only purging it is recommended to have a second schema for the archiving function as process archiving will backup process engine metadata of the models there.

See Monitor Users Guide for how to set up the archive schema.
Remember to place the proper grants in the main schema so that the archive schema user can select and delete in the main schema.

regards,
Holger

You know that operation is destructive, right? That will probably fix your issue but you won’t have anything left if you drop that table. Before you decide doing that, create a catalog for your db and compare that with your webMethods fix level. If you forgot to migrate a DB schema, it may cause similar issues as well.

I’m not going to delete these tables, I understand, it might make the situation worse. In that case, it might be better to install a new instance from scratch. Services work even with this condition, but some things are wrong, e.g. we have duplicate processing orders etc.

I really appreciate your effort.
Thanks

I don’t know which instance you are referring to but still whether its a new DB schema or a new MWS instance, it won’t be easier then recreating tables. You need to change DB connection strings in both MWS and IS. DB is tightly coupled with BPM. If you drop it, you will lose data, if you change it you won’t have the previous data and now you need to update every location unless you migrate it to new db. Try verifying tables using DB component configurator instead. Use catalog option, and compare the versions with the fix level you have in your environment. Read the fix notes, (especially the red ones). See if there is a mismatched version there.

By the way:
Did you try limiting the query to a single instance and see if it returns results?
Did you try restarting archive process? This time keep the task small, limit it to 1 day, and don’t pass the work to DB, keep it in service level and see what it returns.

FYI, if not archiving for awhile, or if this is the first time you trigger it, there is a possibility you are using wrong parameters or wrong service. Its better if you make it a daily scheduled task with default parameters once you get it working.

1 Like