webMethods Mime Streaming with large file

, ,
1

Hello,

I am running into java memory issues with the current scenario:

We have a loop that loops over 25 attachment ids, each attachment is around 100MB. We have 3GB of memory allocated to the integration server itself.

When looping through the ids, we receive the InputStream from our application, pass it into the mimeStream with addBodyPart, and then send the request to an external site. Afterwards, we close the InputStream and then drop any variables that are no longer needed. Once we get to file #13 or so, we get a Java Heap Error because the Integration Server has ran out of memory. Iā€™m guessing there is a memory leak somewhere with how we are dropping variables and such, but having a hard time tracking it down. When I did get the server to generate the heap stack so I could analyze it, it was failing on copying the ByteArrayOutputStream when the InputStream is added to the mimeStream with addBodyPart, but that is probably not the issue. The issue I believe is with the memory slowly being filled up as it loops until it runs out of memory, as I can send a single 200MB file just fine.

Any advice would definitely be appreciated, especially if anyone else has run into similar issues before with large file handling/streaming with mime parts.

Thanks

2

Are you trying to send these large files out as email attachments? If yes, can you zip all of the files and then send out the zip file instead of sending Individual files?

Is the target an SFTP site location? If yes, you do not need to create a MIME object but simply write to the location using the raw input stream.

Cheers,
Akshith

1 Like
3

Sorry, should have provided that extra set of details.

This is transmitting to an external site using the http client as a POST request.

Specifically, we send the file as a Form/MultipartFile as that is what is required by the site we are interfacing with.

4

Hi Logan,

You should try to increase IS memory allocation to 5 GB. As per SAG recommendations it Min is 2 GB, if you are running only one service on this server we can expect there is issue with code.

Kindly provide more details.

  1. Which IS version you are using and fix level?
  2. How many packages your are maintaining in IS?
  3. what is memory utilization when you are not running service?

I hope it will help you.

5
  1. 10.3/IS_10.3_Core_Fix9,IS_10.3_SPM_Fix5
  2. Around 40-60
  3. 2GB used with 1.5GB free. 3.5 Total.

We can try increasing the memory allocation. The main probably is that this is a loop in the process model, and it usually performs fine for about 10 iterations, and then we get out of memory error. So seems to be an issue with garbage collection or a memory leak

6

In the webMethods 10.7 release (coming up this year) there are enhancements to the Integration server mime public services to handle large files. This will not help you now but keep this in mind when/if you work on wM IS 10.7.

7

Thanks, Wayne. We were actually planning to upgrade to 10.7 but since it got pushed back several months, we had to go with 10.5. Glad to hear there are some improvements coming down the pipeline.

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.