I am trying to remove the first line from a large flat file without loading the file into memory. Does anyone have suggestions?
<header>
<record1>
<record2>
<record3>
<recordn>
becomes…
<record1>
<record2>
<record3>
<recordn>
tail +2 <filename>
You can use the flat file schema utility if either the header record or the non-header records have a consistent value you can use as a record id that always appears in the same position.
Otherwise you can read and throw away header data in your own Java service.
In either case, you can submit the file via FTP, File Polling, etc. to avoid loading the whole file into memory. ( IE, use inputstream processing as described in the Flat File Schema manual )
Regards
The file is on my fileServer. Attaining the content in a Stream is not an issue. I use getFile (loadAs stream).
Only the first record has a Record Identifier. I can convertToValues and pull in the file contents as unDefinedData, but this data is now in memory. No bueno.
I’ve looked into the Java Service option, but can’t find a way to make it work with a Buffer. Can you elaborate on your solution?
As an example, when you FTP a file into the Integration Server your service is invoked with an input signature containing an object named ffdata that is an instance of wm.server.net.FTPInputStream which is a subclass of java.io.InputStream ( you can’t do a mark/reset on it ).
You can invoke a Java service with this and do whatever you like with it.
Need more elaboration?
More elaboration would be helpful. I have a handle on the Stream; that is not an issue. Can you elaborate on the Java Classes and Methods that I could use to quickly remove the Header without bringing a 500 megabyte file into memory.
[url=“JDK 19 Documentation - Home”]JDK 19 Documentation - Home
Depending on what you know about the data, you can read() or skip() until the “good” data starts.
Although you don’t want to read really large chunks into memory, you should still read into a buffer for efficiency. You can construct a BufferedInputStream from an InputStream as described:
[url=“JDK 19 Documentation - Home”]JDK 19 Documentation - Home
I.E. java.io.BufferedInputStream myBufStrm = new java.io.BufferedInputStream myBufStrm(myInputStream,4028); //4K buffer
Once you hit the start of the good data, you can start using an OutputStream (also buffered) to write another file.
It would be interesting to hear a bit more about the larger problem you are working on. Is there any mapping or transformation involved?
Oops, sorry.
I.E. java.io.BufferedInputStream myBufStrm = new java.io.BufferedInputStream(myInputStream,4028); //4K buffer