Save pipeline to database vs file

Tom_Rekcut · November 2, 2018, 3:08pm

Greetings,

I need to be able to restore pipeline from multiple executions per same service. I was thinking of storing pipeline in a database, or unique named files.

How to achieve saving pipeline into database most efficiently? Is there any better way than saving to file using savePipelineToFile, then getting a file, and then saving its content as text to DB(I’m using Microsoft SQL DB)? Would You advise saving to DB or unique files, and why?

wayne.leishman.20059 · November 2, 2018, 3:28pm

Hi Tom

When I want to save multiple occurrences of a pipeline for a service, I append a date/time stamp to the pipeline file name. Therefore, at the beginning of your service get the current date/time in a format you want. Then use variable substitution to append the formatted date/time to the file name when you run savePipelineToFile.

Regards
Wayne
Principal webMethods Instructor
Software AG

Tom_Rekcut · November 2, 2018, 3:35pm

That’s what I’m doing at this time, but I don’t like the idea of storing multiple files, so I’m planning to migrate to DB, and I’m looking for a clean way to achieve this

Mahesh_K_Sreenivas · November 4, 2018, 11:17pm

You can store the pipeline data as blob/clob into DB in async logging fashion.

Tom_Rekcut · November 5, 2018, 10:04am

How to get the pipeline data in order to save it to database though?

Only way I can think of is saving it to a file, getting data from a file, inserting it, and then deleting a file, is there anything more efficient?

Mahesh_K_Sreenivas · November 6, 2018, 12:17am

You can get the pipeline IData at the start/end of the invocation by wM Java API.

Baharul · November 7, 2018, 5:21am

I think this is the right way as suggested by Mahesh to store multiple pipeline of service . Get Idata and store as Blob in DB.

But I have one small doubts what is the real-time needs for storing pipeline. So , far I have used only for debugging purpose.

Thanks!

Mahesh_K_Sreenivas · November 7, 2018, 5:40am

Reg. your question on storing the pipeline data, you can replay/retry/resubmit the service (s) at different points in E2E flow. Say you have a child service which is posting data to a target system and if this service fails due to network connectivity or target system down, you can replay the child service with the pipeline data that is stored. It is similar to the service auditing/monitoring on MWS.

Baharul · November 7, 2018, 6:22am

Thanks Mahesh . But I think for replaying of the service service auditing will be the best where built-in features will be handling the scenario , for the child service re-submit is not possible so we can use pub-sub methods where we feel it maybe required.

Thanks
Baharul Islam

Tom_Rekcut · November 22, 2018, 3:38pm

So I found out that IDataXMLCoder class is really helpful in GETTING the pipeline into stream/string etc.

I am saving that in database.

Now I want to create a process which will REPLACE the pipeline with the data I get from the database. I can get it as IData/String or byte array. Is there any easier way to achieve that rather than saving the String from DB into file and then calling restorePipelineFromFile, then deleting a file?

I have tried manipulating both IDataMap and IDataUtil(delete from pipeline, then merge with my IData) with no success. Tried replacing IData objects with no success either. I can’t remove the data from the pipeline, then add it from IData from my database, because some of it’s values are strings, not IDatas, and for them getAsIData returns null