Reading Large Files from file system

, ,
1

Hi,

I have a requirement where i need to read a large EDI file from file system and pass that to the TN using routeXML service.
I can’t use the convertToValues service.

Could anyone please suggest a method for reading the file as a stream and then passing the data to TN.

Quick help is highly appreciated.

Thanks,
Nis

2

you can write a java service using StringBuffer. and get the object in pipeline.

3

Or use pub.file:getFile, with loadAs set to stream.

However, this only handles the first (and easiest) part of Nisha’s post: how to read the file as a stream.

The second part is the tough part: how to send EDI documents to TN via routeXML without using convertToValues. Since routeXML accepts a node (which must be an XML document) as its input and an EDI document is definitely not an XML document, you’ll have to write your own code to convert from EDI to something that routeXML can understand.

Why not just use convertToValue? It works, its modular (accepts multiple EDI document types), and its already built for you.

4

How would the use of StringBuffer help with a large file?

5

Using the getFile will only do when its a small file…i need to read large files using getFile as stream and then i am lost how to handle it without using convertToValues.

6

Before routing it to TN using routeXML…why you want to call convertToValues??

Once you load the file pass the edidata to routeXML thats it.

First try with loadAs (stream) option…

Also dont step thru the flow try Run directly or schedule the service and see how it goes…

Steps:
getFile (loadAs-stream)
streamToString(PSUtil svc) map o/p string to edidata
routeXML(edidata)–send EDI to TN

HTH,
RMG

7

Step 2 will load the entire file into memory, which Nisha appears to be trying to avoid.

8

Do you think any other way to avoid…esp. large EDI files?

convertToValues (Iterator) job comes after routing from TN and hits a service for further processing…Isn’t it??

HTH,
RMg

9

Hello Guys,

If the file is very large and the IS goes for a toss even after processing with ffiterator while convertToValues and or with Stream(streamToString), then I think splitting the file in small chunks and then validating ,routing and processing each chunk can be a way through to avoid loading of entire file. But I dont know how much can this be helpfull to nisha as splitting the the file into many smaller files would break the integrity of the records in file .

Jiten

10

Yes i agree with you…but that is only way dealing EDI large files thru IS service…Incase use of TN LargeFile handling its a straight forward.

HTH,
RMG

11

Just to clarify, streamToString will always load the complete file into memory. When using a stream for large file handling, it is important to not do anything that will read the entire stream of bytes into memory.

12

But with loadAS stream always help dealing performance issues isn’t? BTW,Is there any article that confirms streamToString loads into memory?
just incase.

TIA,
RMG

13

Yes, loadAs stream is the right thing to do to avoid loading an entire file into memory.

But as soon as you do streamToString, it will read the stream until EOF and make a String object out of the bytes. streamToBytes also reads until end of stream.

From the wM docs for streamToBytes: “This service reads all of the bytes from stream until the end of file is reached…”

streamToString is in PSUtilities. You can review the source for that. It too reads until the end of the stream.

14

Thanks for the pointers…I will review the source now…

HTH,
RMG