FTP large files

, ,
1

I have an interface which transfers the files between two applications. Interface is using file polling mechanism. File polling notification will invokes an interface which FTP’s the content to a different application. No transformations or validations to the file content. This is working fine for small files. Problem I am seeing here is source application is generating a master file which is almost 2 GB in size. My interface is taking the stream from file polling notification and passing it to the ftp service as a stream. But this is taking hours to transfer the large file (more than 6 hours). We are using IS 6.5. Can anyone suggest me a better approach to transfer large files across two applications?

Thanks,
Vinny

2

The 6 hour speed problem you might be stuck with.

You can use FTP reget and reput for interrupted transfers. This may not be supported by the ftp server.

As for sending files, you can send the files in packets to the other side using RandomAccessClass package to break up and keep the index. Sending could be done via a wsdl. This would require the other end to reassemble the file, ie have a client application.

3

Can you share details of how you are reading the stream and writing the data to the target?

4

Hi Vinny,

When you have large files to be transferred (~1GB or more), don’t use file polling to load the content to webMethods and pass it to FTP API. As it loads the file content as a stream to webMethods JVM.
Rather use mget and mput service to transfer the files, where the files will be transferred using native ftp client and the content will not be loaded to webMethods IS.

Cheers
Subhra

5

Implied in your post is that reading the file via stream loads the entire file into memory at once. It does not. pub.client.ftp:put/mput use streams.

Using IS to read the stream isn’t a problem in and of itself. It is how the stream is read (and buffer sizes) that will impact speed. That’s why I asked for the details of how the stream is being processed. With that we may be able to determine changes that can be made to speed things up.