I’d like to ask for your guidance regarding processing of large XML files in flow service. Right now I need to deal with files around 30-50 MB which are sent via FTP to WM IS.
I have the following process:
XML file --ftp–> IS service (wm.tn.doc.xml:routeXML) —> TN (document recognition and query for information from the file then triggers processing rule) —> IS (mapping to new XML format & validates XML) —> TN —ftp–> Partner
Structure of the XML file is like:23232 27000 2015-03-03 20000 FRST NORM ... ... ... ...
PmtInf can have 1 - N occurances
CdtrTrxInf which is subnode of PmtInf can have 1 - N occurances (e.g 20 000).
I need to process this XML file to map it to a new XML format then validate it against schema.
I know that for better performance I could use getXMLNodeIterator and getNextXMLNode. But with the XML structure like above I can only use movingWindow=true only on first level to get GroupHdr and PmtInf. Once I am in the PmtInf node I can use getXMLNodeIterator again to iterate over nodes CdtrTrxInf but with movingWindow=false. This is probably not the best solution and doesn’t provide all benefits from using getXMLNodeIterator with movingWindow.
Iterating through all those nodes I would map all information coming in xml file and create a document in memory having a new structure. Using this approach I would create a quite big document in memory (by adding processed nodes via pub.list:appendToDocumentList) which I need to convert to XML string (pub.xml:documentToXMLString) and then validate (pub.schema:validate).
Could you please advise how this type of XML file can be processed efficiently in WM flow service ?