Thanks for the follow-up on how you resolved the issue. Most folks don’t do that but it is quite helpful for future readers.
Do you close the stream on the IS side? You might also drop everything in the pipeline at the end. That may help (or may not) with the multiple files per session.
Yes, I do close the stream and drop all items at the end of the service.
I did find some documentation on the FTP inbound stuff. It’s in
the Flat File Schema Developer’s Guide. Not a lot of detail but enought to point in the right direction with regard to the content-types, etc.
_brett.
I did a test and it worked (uses ffdata since my file is has .txt extension)
Find attached the sample service SandBox.Nancy:testForum (You will need to create the package first since this is an export from developer)
Also find enclosed the pipeline file showing the output. Let me know if this doesn’t help.
Hi Brett,
Not completely sure what the case is you are trying to solve, but I think you may want to consider your architecture. If you are receiving binary files, and (as you said) you would like to receive multiple files concurrently, I think you should try to write the file to disk asap once received, to ensure the release of the thread. Once on disk (even in a temp dir) it will be much easier and more efficient to process the file. Additional benefit: includes archiving.
To write a file to disk from the FTP input, you need a java service that takes any sub class of Inputstream (aha, such the FTPInputStream), and writes it to a file location.
Something like this:
// Get params
IDataCursor pc = pipeline.getCursor();
InputStream s = (InputStream)IDataUtil.get(pc, “contentStream”);
String f = IDataUtil.getString(pc, “filename”);
// TODO: check whether file is writable, etc
// Try to write stream to file, fiddle with the buffer size for performance
try {
FileOutputStream fout = new FileOutputStream(f, true);
int pos = 0;
int read = 0;
byte buffer = new byte[8192];
while ((read = s.read(buffer, 0, buffer.length)) > 0) {
fout.write(buffer, 0, read);
pos += read;
}
s.close();
} catch (Exception e) {
throw new ServiceException(e.toString());
}
// Output nr of bytes written to file
IDataUtil.put(pc, “bytesWritten”, String.valueOf(pos));
pc.destroy();
After running this, the stream is closed and can be dropped. If you would use a broker to submit a notification to a trigger, the current thread can return a confirmation to the FTP client, and a separate thread can start the processing of the file. Setting the trigger to concurrent easily allow parallel processing.
Hope this helps!
Chris
I’m not sure when it was added, but 6.5 has the ability to write an incoming file to disk and publish an event without needing to write a service to store the contents.
In the IS Administrator’s Guide, look at watt.server.userFtpRootDir in Appendix B. Server Configuration Parameters. It describes how IS can behave more like a “real” FTP server, storing the put file to disk and publishing a “putCompletedNotification” rather than invoking a service.
True, but as I understood from brettp’s posts, he really wanted to manage the byte stream, doing 4 bytes at a time, for whatever reason, I assumed he would be past that station.
If that is not the case, separating the native file handing for FTP as standard funcs would be the best option indeed.
Chris