We have developed a file integration mechanism whereby we FTP.get the files from other systems, put them into a polling folder, thru the filename decide whether processing needs to be done on the file and invoke the processing service and then ftp.put the files again to other systems. All these are done just using the IS.
It seems to be working fine for awhile but recently some of the files polled for processing seems to be truncated. We are not sure if its truncated at the polling portion (when getting the ffdata), or was it truncated during the convertToValues (when the ffdata is converted into a doc). The worse thing is that this happens intermittenly and we can’t recreate the error (when we test with the same file again it ran fine).
The files are not that big, only about 5 to 8 MB, and truncation can happen at only 500K (meaning 450 records out of about 5000 records came thru). There were no errors, except that the files were put into the polling error folder.