Large document handling with out TN

Hi all! I have an integration that was designed to handle large volumes of small documents. The integration receives PAYEXT IDOCs from SAP into partner manager, the trigger service picks up the IDOC and maps it to our Payment canonical, publishes it to the broker, subscriber on the external IS puts the canonical into TN, TN processing rule triggers the mapping service to create the externally facing format (in this case EDI X12 820 document), envelope, deliver to TN to a public queue for delivery to the trading partner. This integration was designed for small documents and immediate delivery.

Unknowingly our users have been in the process of setting up a new vendor to pay thru this integration. This new vendor is a monthly run that creates a huge PAYEXT IDOC. Our typical size is 17k and this IDOC delivered to our production system was over 27 megabytes. Of course it had a major impact on our system when they submitted the IDOC. Our estimates on runtime were 3-6 days with the way it was running before we cancelled the job.

So now here is my dilemma I have only written integrations that handle frequent, large volumes, small size documents. How do I in a single integration handle both small and large efficiently? The initial receiving of this large document is on our internal server that does not have TN so I cannot use TN large document handling functionality to do this, at least not for that piece of it.

Any hints, tips, and/or pointers on where to go to research how I can accomplish this task efficiently will be much appreciated.

TIA,
Dawn

If you are not able to use TN large documnt handling then may be you can try using the NodeIterator and NextNode services in the IS for parsing document by chunks to save memory when processing translation mapping to cannonical.

http://www.wmusers.com/wmusers/messages/100/36914.shtml

Please search on this site,there are quite number of threads discussed on this largedocuments handling/processing Architecture.

Thanks rmg! I’ll do some more searching. When I searched on large document I got mostly the TN stuff. Thanks for your input!
Dawn

Also search on NodeIterator…stuff too.if that helps.