Large Flat File Handling

, ,
1

Hi All,

Can any one help me on Large Flat file handling mechanism.

Currently i am trying to handle the large file by converting the Flat file to xml data and using the Node iterator.

But i am getting the Outofmemory exception while converting the document to string
I am using the documentToXMLSting service by providing document and documentTypeName fields.

Thanks,
Nagendra

2

Hi,

What is the size of this large document you are trying to process? what is the max n min Heap size u allocated to your IS?? Also check the keepAliveTimeout value in your environmnet… these all parameters should be enough to handle the load you are putting on IS.

cheers!
nD

3

Hi,

  1. First find out how many transactions are there in the file (suppose say 500).

  2. Decide how many transactions you want to process in one go (say 100).

Repeat Specified number of times ( here repeat count is 4 )

Keep these steps under repeat…

  1. Create a java service which will read 100 transactions at one go. This java service can give the output in form of string.

  2. Pass the string which you receive from customized java service to convertToValues to create IS document.

  3. Map The Canonical to Output file format…

4

“But i am getting the Outofmemory exception while converting the document to string”

It sounds like your steps are something like this:

  1. Use node iterator to get one record from the file.
  2. Convert that to a document and append to a list.
  3. Repeat until all records read.
  4. Convert the resulting document (that has all items appended) to a string.

Does that match what you’re doing?

If so, then the problem is that you’re effectively loading the entire file into memory. And then duplicating it when trying to convert it to a string. So if you have 100k file, you’ll have 100k+ document representation in memory and then try to create a 100k+ string in memory.

When dealing with large files it is important not only to iterate over the source, but to not create the entire target in memory either.

If the steps above are not what you are doing, please provide additional detail so we might see where memory is being exhausted.

5

Most likely this is the problem. It is not sufficient to only stream the input document. When dealing with a large document transformation it is important to stream the output as well.
One way to do it is to output the generated XML into a file in chunks.

6

Hello All,

I am new to webMethods,I sending a large flat file but it does not send it shows an error like java.lang.OutOfMemoryError: null.Please help me any one.

Thanks&Regards,
Prem sai