I’m sure that you can do that, but I am not sure what the result would be since I have not tested this specific functionality outside of either a single-server instance or a clustered instance.
In a clustered instance, each machine in the cluster is a node belonging to the application space so to the application, it appears as one machine. Also, the repository kicks in to persist run-time variables between nodes.
I wonder what would happen if one box greps the directory for files almost simultaneous to another. My question is would you generate a file not found error?
Another thought: you could possibly try this if you staggered the file polling between boxes so that no box polls at the same time. I would seek to synchronize the machine times for this to work correctly.
One other aspect that requires substantial thought is the processing time of LARGE documents. I have seen file polling used when processing large data dumps where the partners have FTP’d the files into the system and IS scans and processes. Make sure that you use working directories because IS will actually move the file out of the polled directory into the working directory.
If this is a production instance, I strongly suggest clustering the boxes since it will provide the failover normally required.
Load balancing normally either round-robins or monitors response times and delegates the request to the server with the quickest response. But does not provide for failover.