Is there any inherent file size limit in webMethods as to what can be processed? I work at a Fortune 50 corporation, and we are receiving EDI 835 files from a large national healthcare payer that are over 2GB in size, all of which are contained within a single ISA/IEA envelope. My webMethods team (offshore contractors) are telling me there is no way to process files larger than 2GB, and I am doubting their claim. Hard to believe that this well-established healthcare payer would produce 835 files that can’t be processed ‘ever’ by webMethods…
If files larger than 2GB can indeed be processed, can you point me toward the right place in configuration to change that limit?
Irrespective of Single or multiple ISA, You might need a larger memory to process the GB’s of files.
ideally, Creating chunks out of the large file will give you better preformance rather than loading entire files in your JVM at a time.
Also, make ensure to process these files during non-peak traffic if it is scheduled once. if not process these files in a separate instance would be best as not to put overhead on your server resource.
Memory is a resource that needs to be handled efficiently. In your case, increasing memory is an option but the problem will return when your file size grows.
We have a solution that will process 50+GB files on a 1GB memory footprint within minutes without ever overflowing the heap memory. The solution is called DataFeedr (datafeedr.io) and runs on top of Integration Server. If you want more information, please reach out. My email is christian.schuit@centipod.nl
Our solution is a file processing platform that runs on top of IS and handles large files. All other IS functions are still available. For instance, if you are using Trading Networks, you can still use that for processing the EDI content in the same manner you do now.
A possible solution could be to configure DataFeedr to pre-process the EDI file and break it into smaller sets of 385 documents in smaller envelopes (or even singles). To confirm that all data is processed, we build up a reconciliation report while processing and send that after processing completes. Such a reconciliation report can also be processed automatically to find inconsistencies. The smaller envelopes will go to TN for regular EDI handling. And TN can still handle the correlation with the relevant 837’s.
If errors occur, the erroneous 385’s will be kept separately for inspection and possible resubmission. Or an error can start an alternative processing route or even a BPM process.
Our solution allows full control over memory and CPU usage so that you can process these files even during peak hour. And if your file sizes grow, that will only impact the duration of processing, not the memory or CPU usage.