query a customer erp system for item or customer master data
loop through the results and build an xml document (just a long concatenated string)
transmit resulting xml document to the target system
My problem is that I can’t make it perform. The sql query is fairly fast (maybe 1 minute), and the transmission is fast as well. However, the looping through and putting into an xml string is very slow. For about 40,000 records with about 24 elements each, it takes a little over 24 hours. I must be missing something. We are using IS 6.1 on a fast windows server (3GHz) with the maximum amount of memory. Any suggestions would be appreciated.
Have you tried picking up the result set and directly converting to XML as is from the IData Structure? How much time does that take. This XML Structure may not be what you expect, but it will still be an XML.
The next thing to try is, if you have an XSLT that you can apply on the generated XML to create your required XML, and perform that using Java or WmXSLT, what time would it take?
I’m just guessing at your code, but if you’re doing 40000 individual string concatentations, and assuming that each customer record produces an XML fragment of N characters, then over the course of the loop, Java will need to allocate 800,020,000 x N chars of storage. That will take some time, and put a big load on the GC. 24 hours worth, even
(BTW, the same sort of issue applies to incrementally building string or document lists, as WM implements them internally as arrays, and creates new copies when the list is extended.)
You’d be better off creating a single StringBuffer of 40000 x N chars, and appending the XML fragments to it, or breaking the extract into multiple parts. You could wrap the StringBuffer in some services to allow it to be used in Flow, or you could just do the whole construction job in a single Java service.
