I read a thread on this, but there is no solid solution. I have some question and suggestion here:-
Objective:- To append two large file back to back. (File size 100MB - 350MB)
IS Server Heap Size:- 1 GB
Problem:-
(1) The flow service-getFile causes “out of memory” problem.
(2) Before hitting this flow, the used memory is 510000KB
(3) After hitting this flow, the used memory is 800000KB
Quesions:-
(1) The increase in memory usgae is 300000KB = 299MB, for a 200MB file processing, is this acceptable?
(2) If the heap size limits the size of flat file IS can process (append, read, modify, insert etc), and the version of JVM limits the Heap Size you can set (1.4 max heap size is 2GB?), one can actually predict the maximum size of file IS can process
(3) Can the append action be accomplish using System memory instead of Heap Size?
(4) What is the different between getFile as Bytes and Stream?
TN has some facilities for handling “large files”. Unfortunately, the best strategy for handling them are not to read them all into memory 8(…
Specifically, there are limits on how big some objects can be (String?), and some built-in services cannot handle horrendous data sizes. Normally you can just submit the large data to TN, and have it return to you a “bizDoc” with the large data stream in “contentParts”. But this doesn’t quite seem to fit your requirement.
So. If your requirement is just to add the files together (or should I say concat), w/o having to processing the final result in memory, how about just calling the OS to do it (UNIX: “cat file1 file2 > file3”, Win: “Copy /B File1 + File2 File3”)?
Answer to some other questions:
2) Max heap size depends on the OS platform and the specific JVM. This is actually a tough question. Haven’t seen a place where somebody puts out a nice table of Platform vs JVM vs MaxHeap… Don’t think 1.4 on Windows is going to be any more than 1.6GB.
3) Not likely. But with the suggestion above, actually little memory is required