Hi,
We have developed a file integration mechanism whereby we FTP.get the files from other systems, put them into a polling folder, thru the filename decide whether processing needs to be done on the file and invoke the processing service and then ftp.put the files again to other systems. All these are done just using the IS.
It seems to be working fine for awhile but recently some of the files polled for processing seems to be truncated. We are not sure if its truncated at the polling portion (when getting the ffdata), or was it truncated during the convertToValues (when the ffdata is converted into a doc). The worse thing is that this happens intermittenly and we can’t recreate the error (when we test with the same file again it ran fine).
The files are not that big, only about 5 to 8 MB, and truncation can happen at only 500K (meaning 450 records out of about 5000 records came thru). There were no errors, except that the files were put into the polling error folder.
The way I’ve attacked these problems in the past is to include validation information with the transmission. You can prepend the value to the start of the file, or you can create a second file that travels with the original. The value can be a count, sum, checksum, whatever. At each potential problem source, compare the file contents to the validation value. This will allow you to identify the source of the error.
I agree with Mark’s valuable suggestion,see if it helps for finding the rootcause.But i have not seen this truncation behaviour anytime during filepolling.
If problems still persists,open an SR with tech support and let everyone know the resolution incase they provide.
I have seen this happen where an FTP was in progress when the polling kicked in and launched the flow that was to read the file. The file was not completely sent before the flow was activated.
You can increase the time between polling to fix this (sometimes). Does your .get command run at a specific time? How long after the get command runs before the whole file is transferred? What is the most time it could take? You can look at some network monitoring tools to find when peak network usage occurs that could slow down the transfer. Try to schedule your get commands during the periods of low network usage. Usually 8AM and right after lunch (1PM) are peak times as people are logging in (all about the same time).
It is hard to recreate the problem because when you see the file in the directory, the transfer has completed. Try launching the flow a few seconds after you issue the get command on the 8 mb file.