Large XML file (ISO20022) processing in flow service

, ,
1

Hi,

I’d like to ask for your guidance regarding processing of large XML files in flow service. Right now I need to deal with files around 30-50 MB which are sent via FTP to WM IS.

I have the following process:
XML file --ftp–> IS service (wm.tn.doc.xml:routeXML) —> TN (document recognition and query for information from the file then triggers processing rule) —> IS (mapping to new XML format & validates XML) —> TN —ftp–> Partner

Structure of the XML file is like:

23232 27000 2015-03-03 20000 FRST NORM ... ... ... ...

PmtInf can have 1 - N occurances
CdtrTrxInf which is subnode of PmtInf can have 1 - N occurances (e.g 20 000).

I need to process this XML file to map it to a new XML format then validate it against schema.

I know that for better performance I could use getXMLNodeIterator and getNextXMLNode. But with the XML structure like above I can only use movingWindow=true only on first level to get GroupHdr and PmtInf. Once I am in the PmtInf node I can use getXMLNodeIterator again to iterate over nodes CdtrTrxInf but with movingWindow=false. This is probably not the best solution and doesn’t provide all benefits from using getXMLNodeIterator with movingWindow.

Iterating through all those nodes I would map all information coming in xml file and create a document in memory having a new structure. Using this approach I would create a quite big document in memory (by adding processed nodes via pub.list:appendToDocumentList) which I need to convert to XML string (pub.xml:documentToXMLString) and then validate (pub.schema:validate).

Could you please advise how this type of XML file can be processed efficiently in WM flow service ?

2

Can you split this XML in to smaller XML files with about 100 payments in each file? For this you can either use Java SAX or StAX. You might have to change the NbOfTxs field to reflect the exact amount of transactions in each file.

3

Hi

Thanks for your feedback.

Unfortunately I can’t split the file into smaller chunks.

For now I try an approach with checking if file coming form TN is LargeDocument then accordingly handling this.

For Large XML file:

  • extracting data from file from TN using wm.tn.doc:getContentPartData as file is store in tspace. Then converting this to string and looking for certain XML tags.

Don’t like this approach too much. Not sure how can I use NodeIterator on the file coming from TN stored there in tspace.

Is this a better way to handle Large XML file ? Would it be use of Java Stax ?

Thanks in advance for any help.

4

Maybe that property is inherited to child iterators also … Can u confirm u raising a SAG ticket.

5

Maybe that property is inherited to child iterators also … Can u confirm u raising a SAG ticket.

6

mrbungle,

Yes you approach via TN and check the flag is doable and it should work unless you have configured TN large file handling configuration and tspace threshold etc…

HTH,
RMG

7

mrbungle, did you finish this task ? Any issues so far ? Kindly share your observations on this issues.

Thanks,