TN, EDI module and RosettaNet module all support large document handling. Each of these provide documentation on configuration and development.
Unfortunately, handling large documents isn’t simply a matter of configuration. As several people have suggested in various threads, increasing the memory allocation to the JVM can sometimes address the problem, depending on a variety of factors (physical memory, garbage collection, document size, server load, etc.). Increasing the JVM memory moves the out of memory error to a higher number, but at some point you’ll likely run into the problem again. So…
The basic approach to handling large documents is to stream them to disk as they come in, rather than storing them in memory, and then passing an InputStream object to a service for processing. TN does this based on doc size. The EDI module does this based on content-type. This means that the normal stringToDocument, documentToRecord, map to another record, recordToDocument approach isn’t sufficient–this technique loads the entire document into memory.
So you have to process the stream in chunks. There are built in services to help (node iterators, content part reads, etc.). Basically you’ll need to read in a block of data, convert that block as needed, and stream it to some location (no sense in creating the target all in memory–you’ll run into the same problem you’re trying to avoid on the input side). Needless to say, this makes mapping a bit more complex than the normal “small” document handling.
The wM docs are a good starting point for understanding how to do the work. It’s not trivial. And you may end up needing to rework a good portion of your existing integration code to accomodate large documents.