File Poller picking up incomplete file

, ,
1

Hi All,

I have a situation wherein a service has not completed writing a file and the poller has picked up the incomplete file. Is there any work around for this.

I am using the PSUtilities:file:writeToFile for writing to file, is there a bug with this service or is it a bug in the File Polling port of webMethods.

Thanks
Gaurav

2

If you are polling a directory for a file with a certain name or extension, have your service that is writing the file re-name it after writing is complete.

3

This is definitely not a bug, but an incorrect understanding of file handling. As Chris points out, you need to make sure that the file polling process does not try to pick up a file before it has been completely written.

A common approach is to write the file using a particular extension, for example myfile.tmp. When the writing is complete, then rename it to something like myfile.dat.

Configure the file poller to pick up *.dat files and the picking up a file prematurely problem will disappear.

4

Or…you can make use of the File Age field in the file polling port. Only pick up those files which are x seconds old. This will ensure incomplete files are not picked up.

5

It makes it more likely, yes, but it does not ensure it. Factors that impact the time it takes to write the complete file, such as a large file or any network latency, can cause the problem to arise. Another scenario that can only be solved by using the rename technique is the case where the file is partially written but the sending source fails before completion. This will also cause a partial file to be processed.

IMO, the file age approach to picking up files should be avoided as it doesn’t solve the problem–it just decreases the possibility.

6

I would agree with Rob. We used the same policy.

Poll for a file with *.txt extension.
After picking, rename it to *.txt.working.
Once the writing is done, rename it back to *.txt or *.txt.done

My 0.02.

HTH

~Malhar

7

We use two folders in this situation where the file with data written first into data directory, once write operation is completed then write a zero byte file into trigger directory with the same filename. Then we configure our adapter to poll trigger directory first, use the filename to read the data from data directory. Our folders will look lik the following

/data
/working
/completion
/incoming (or) /control (or) /trigger

This worked for us in the past. (We are still on very old version of wM)