Hi, In one of our implementation we have to transfer the files from one FTP server to another FTP server using webMethods. I was going through the ftp BIS and it says we can specify “largefilethreshold” to read a large files. But do we have any constraints on the file size that we can read and put to another ftp server using webMethods.
When we try to read (get) a file of size 20 GB, does this operation consumes any huge memory in wM. Will this affects IS performance?
What will be the right design to read a large file and transfer it to another FTP server through wM? (we need to do plain transfer…not to parse the data).
(Eventhough wM is not a right FTP client to transfer files from one ftp server to other, we need this to do it as we can actively monitor the transfer with monitoring team we have)
If you need to handle such huge file, you can’t keep it in memory, you have to stream it and save it in the local hard drive.
You can follow the instruction on how to handle large file in WM (WM also save it to disk). The build-in service pub.client.ftp:get, pub.client.ftp:put already has stream parameters.
Or write some simple java code using: FileInputStream and FileOutputStream.
This require very small memory footprint and you can handle almost any size doc. keep in mind increase the connection timeout limit to allow the download/upload to finish.
Forget about proxy’s, this is a FXP transfer and IS would be the “client” to launch the command without any control. Has lots of problems between firewalls and both FTP servers must support this.
I don’t see the need to use IS to transfer a 20G file if you do not want to use this file for parsing it.
Yes, you and I may not see the need to use IS in this scenario. If I had the choice, I wouldn’t. Ravindar also already acknowledged the fact that using IS is not the best solution. This was the requirement he was given nonetheless.
Thanks for your feedback on the proxy command though. Can you provide a little more detail on the firewall issues you’re referring to though?
I agree, and previously stated, that this command will not work on all FTP servers. Now, as to your point regarding the IS not having any control over the transfer, I’m not sure I understand. The command is issued by the IS and the IS will receive a response indicating whether the command succeeded or not. Of course, the transfer will be done from FTP server to FTP server directly, but that is the whole point.
On the other side, IS is the client and does only have control over the “start” and “end” commands, would not be aware of status (percentege of transfer), transmission speed, and of course cannot be used to reconnect, ressume etc, so that’s all.
Remote FTP server needs to have all ports (or a range of high-ports for PASV defined) and enabled on the firewall, same for Source FTP server. Behind NAT this usually fails.
FXP is great! I love that protocol and used for years, however I do not recommnd it due the complex setup in today’s enterprise firewalls.
If you talk about public FTP sites, then go for it.
Still your POST give a good alternative in case this is a must, and indeed it might be the only solution
Use get to retrieve the file from A. Specify 0 for the largefilethreshold and null for localfile. This will write the content to a temporary file and return an input stream.
Logout of A.
Login to the target FTP server (B).
Use put to put the file to B. Pass the stream returned by the get in step 2.
Close the stream (just to be safe in case put doesn’t do so).
Logout of B.
Remove/archive the temp file–I’m not sure if IS automatically deletes the file on close. If it doesn’t, then you’ll want to use an explicit dynamic filename in the get in step 2 so you can move/delete it later.
Of course check errors after each step.
Be sure to use transfermode set to binary if appropriate (usually the case).
