Processing large files

, ,
1

Hi,
We have a trading partner who is going to send us large files( 50MB to 150 MB). we have a custom envelope in which they wrap this file and FTP it to us. The problem is we have to seperate the header and trailer from the actual data. I preferred to write java service for this but as the java service degrades the performance is there any other alternative for this.
Thanks,
Jay

2

Jay,

One option would be to create a custom contentHandler that both streams the data to disk using Java InputStream objects and strips off the header and trailer information. The temporary filename, header and trailer info could be passed to the invoked service.

There are examples on Advantage and here on wMUsers of custom content handlers. These are written in Java and registered as a startup service in a package.

Mark

3

Mark,
Should the header and trailer need to be of fixed length for the custom contenthandler to handle the file, because our custom envelopes header and trailer are of varying length.
Jay

4

Jay,

I guess there are really two parts of this task. 1) get the file to a server that is accessible to IS 2) process the file.

The custom content handler approach would help with #1 assuming you want to use the built-in webMethods FTP server. If you use the standard IS FTP process to process large files, you will quickly fill up available memory in your server and die an ugly death.

You could accomplish objective #2 using the flat-file processor once the file had been streamed to a temporary location on the file system or you could parse out the header and trailer records in your content handler if your java skills were up to the task.

Mark

5

Thanks Mark for your suggestion. We don`t use IS FTP. The file is pushed by our partner to our server. We just have to seperate the header and trailer and i wrote a java service for that.
Jay

6

You could also use the services in the WmFlatFiles package to do this without java.

Mark

7

Mark,
I tried to use services in WmFlatFile package but they didnot satisy all the needs for this case so i opted to write java service.
Jay