how to read from a file record by record

, ,
1

Scenario: We need to read from large file, which contains records delimited by /r/n.

Problem: I know of way to read using a java service where you pass on the Reader object from pub.file:getFile to this java service. Which then uses readLine method of BufferedReader to read record by record and call further flow service with this data. But my question is “is there a way to do the same using webMethods built in services” ?

2

The flat file Adapter is able to process the streams by node or by record (depending on your flat file structure). Please see the section “Handling Large Flat Files” in the “Flat_File_Schema_Developers_Guide” for details.

Regards

Martin

3

Thanks a lot!

Regards,
Srivatsa

4

if you want to read in incrementally due to memory issues (of a large file) also think about your output. If you are writing out to another large file you are still likely to have memory problems. You may also have to write out incrementally as well.

5

Yes that’s right. For the same reason I have used Stream on both input and output.

I am using Stream option in getFile and then using a FlatFile convertToValues service to parse records(set iterator variable to true) and on the output side using pub.io:stringToStream service and then pub.file:streamToFile option.

6

Are you doing this:

loop
…read data from source file
…convertToValues
…map it
…convertToString
…stringToStream
…streamToFile

If so, you can eliminate stringToStream and simply use stringToFile. stringToStream simply converts a string to a byte array accessible via the ByteArrayInputStream interface. It doesn’t provide any value in this case.

7

Oh ok that’s exactly what I was doing! I will change it to stringToFile then. I was trying to use stream function similar to java output stream(though I couldn’t find a Output stream service) thinking that stringToFile would be a little slower compared to stream.

Thanks for the clarification Rob!

8

stringToFile might be a bit slow, as it will open the file, write, then close each time. But this speed may not be an issue for your scenario.

If you find, however, that speed is a factor, then you’ll want to consider managing the writing of the data to the file yourself. Alas, the built-in services don’t provide any assistance for this. You’d need to create Java services to open a file and return an OutputStream-based object for your use. Then a service to write that stream. Then another to close it when done. There are services in PSUtilities that can be copied to your own “public” or “util” package for this purpose.

9

I guess I would go with your suggestion. Thanks for the help Rob.