We have a 3rd party package that communicates via files that it places into an ‘out tray’ directory which is polled via sFTP or it receieves via an sFTP to it’s ‘in tray’. I now want to use webMethods IS to replace the existing interfaces so that the information can be published to or received from a number of systems using the ESB.
My question is what is the best practice approach to poll a remote server for files using webMethods. From my reading so far I can conceive a few options with what I think are the pros/cons/questions:
Regularly schedule a flow service that uses pub.client.ftp:get to attempt to sFTP the file and process it.
Pro: no extra infrastructure
Con: what if the FTP fails or fails halfway
Con: what if the service fails after retrieving the file
Q: How would we set up the secure connection?
Regularly schedule a flow service that uses pub.client.ftp:get to move the file to a directory the IS can ‘see’ and use a file polling port to process
Pro: better handling of the service code failing (file polling will move to error directory)
Con: Needing to set up storage for the FTP to write to
Use an FTP service to poll the server and transfer to a directory the IS can ‘see’ and use a file polling port to process
Con: more complicated and need to rely on other infrastructure
Pro: FTP service would likely have better error handling
Pro: webMethods IS implementation is simpler
Please can anyone advise on the best way to do this? Don’t be limited by the options above they are just my best attempt at figuring out some options
May I ask for your IS version? Before 9.5 of 9.5 and newer?
Background:
Before 9.5 the IS only supports FTP and FTPS out of the Box.
Starting with 9.5 there is support for SFTP, but this is not available for FilePolling-Port.
See Built-In-Services Guide for pub.client.sftp:* servivces
SFTP can be configured in IS-Admin. See Administrators Guide for Details.
One option might be not to poll for data but have the partner tell you about the availability of new data by using a WebService for this. After that you can retrieve the data by normal flow implementation.
Thank you for your reply Holger and good point about the version, I should have included that, we are using 9.5.
Unfortunately we are not currently allowed to change the way that the other system behaves and for future interfaces it may not be possible to change the other system so ideally I am looking for a way to achieve file polling of a remote directory using SFTP.
From what you say if FilePolling-Port does not currently support this I can either poll using a regularly scheduled call to pub.client.sftp:get or use a FTP client to transfer the file securely for the IS. I’ll read up on pub.client.sftp:get to check the error handling makes the service easily supportable.
@Martin thank you for the suggestion, we are not licensed for that unfortunately so we might need to do something else at least for the time being.
@RMG I am leaning slightly using a dedicated SFTP client to transfer the files, I am concerned managing a lot of remote transfers via scheduled services in the ESB could be a bit tricky. Has anybody tried it this way who can put my mind at rest? Is a combination of IS admin and MWS sufficient to deal with any file transfer failures etc?
Yes with IS services in 9.x sFTP (configure alias) and schedule services would not be an issue to manage and proper error handling can be built with in the flows…