We are using the OpenSSH package downloaded from Advantage to sftp files from a remote server to the filesystem where the Integration Server is running. We are beginning to frequently experience an issue where the sftp commands will fail with the following error:
com.wm.app.b2b.server.ServiceException: I/O Exception running command: java.io.IOException: Not enough space
After some research, we determined that the error was being caused by a lack of physical memory on the server where the IS is running. Killing some processes that were consuming a large amount of memory allowed the sftp commands to execute successfully.
In researching the issue, however, we observed that the IS executing the sftp commands was consuming a large (3GB) of memory. We have java startup parameters limiting the memory of the JVM to 512MB (-mx512M). It appears that the OpenSSH is forking new processes that are allowing the Integration Server to exceed the memory limitations we have tried to impose with jvm startup parameters.
Has anyone seen memory issues such as this associated with OpenSSH before?
Are you sending files from or to IS? Which version of IS? How large are the files that you are attempting to send via SFTP? If the SFTP services work like the normal FTP services do, files received are loaded into memory until fully transfered and then the file contents are passed to the processing service.
Agree. Files that small should not consume your IS memory. Not sure whether the OpenSSH example package is supported, but you might see if Support will give you a hand.
I think, WM OpenSSH pacakge executes shell commands using Java Runtime.exec (WM CommandRunner). This would cause IS to fork off child/ren with same resources as the parent (IS).
Not enough space error is probably result of running out of file descriptors due to forks from IS.
We also have a problem in which sftp using openssh is not happening at times. Can someone tell where the error message “I/O Exception running command” will be written. We have the nohup.out file and the IS server log file being written. But we could not find the entry there.
We are also facing the same issue, but we didnot have this issue in Dev and UAT. We are facing this in Production. can you please let me know if there is a solution or work around for this issue.
I did an implementation with OpenSSH “hostkey” based authentication and everything went fine, I was looking for a way to have SFTP with UserID/Password instead of host key,
and I got this java code form [url]http://www.jcraft.com/jsch/[/url] it works fine for me based on hostkey; ( and I manage to get it working with pwd too by calling shell script since it was killing the server-and I still working on calling directly from IS-).
So as an alternative you can use this package to implement SFTP hostkey based authentication instead of the OpenSHH.
Here’s is what I need to do:
SFTP Get a 10-15GB large file in webMethods 6.1, convert it from ASCII to Binary (and vice versa) and SFTP Put to another system.
Pls advise what would be the best way to do this? I’m thinking of using OpenSSH pkg but worried of overloading Pipeline Memory if IS tries to keep the file in memory while picking it. Pls help!
The OpenSSH utility spawns a new thread with memory equal the the JAVA_MIN_MEM (-Xms) parameter set for you jvm. As long as that value is enough to run the sftp from an OS standpoint, you’ll be fine. The contents of the file are not loaded into the pipeline. The memory requirements to sftp the file should be much less than the 10GB file size.
Thanks a ton, Brad. But do you think it can handle a 10GB file even if the JVM is set to, say 1GB?
Secondly, once I get the file and place it in a folder on IS I need to invoke some Unix commands to act on these files - e.g. dos2unix. I was thinking of using a wrapper to invoke a Unix script which does this. Is there any other way to play around with a 10GB file?
Thanks Sudheer. But chunking would involve a lot of exception handling needed. If I make use of wrapper to invoke the Unix commands directly, wont it be good?
Our requirement is that we should have different RSA key pairs for different SFTP partners, so that if we want to change one of the keys the other partners are not affected.
I didnt find any reference of creating multiple key pairs for multiple hosts in the OpenSSH PDF. Can someone throw some light on this please?
