What product/components do you use and which version/fix level?
webMethods
Are you using a free trial or a product with a customer license?
licensed
What are trying to achieve? Please describe in detail.
Trying to replicate files from sftp server to s3, have 2 questions with different scenarios:
how to replicate only new files, any comparison mechanism? I am looping through the list that contains all the files in a remote folder, but it is replicating previously replicated files as well every time the job is triggered.
how to move files from sftp to s3 instead of copying/replicating. Since there is no “move” SFTP connection method I assume I should just delete the file after the replication?
Do you get any error messages? Please provide a full error message screenshot and log file.
You’re on the right track with copying the files from SFTP to S3.
Options to avoid repeatedly copying the same files:
Delete the source file, if no longer needed
Rename the file to another directory (which is a move)
Keep track of the latest “last modified timestamp” seen for any files that have been copied, then ignore any before that.
Which webMethods product are you using? There are several products under the webMethods umbrella. If Integration Server, you can store the “last modified timestamp” in a couple of different places so the each run only picks up recently added/updated files.
[Edit] The 3rd option above is really a “last resort” option. It can be error prone with a slight risk of missing a file. If it is at all possible, either move or remove the file after successfully retrieved. And even with that, try to make it such that it is okay if a file is copied more than once – either notice that the same file (with same attributes/timestamp/size) is at S3 and skip the put, or just overwrite.
Thank you! I use webMethods integration. One more question related to this. I cannot upload my excel files to S3 from sftp. After loop & download file, I use amazon/S3 Upload File.
Upload file " Content" spits out a file with filepath information as a content (which is logical).
Upload file “Stream reference” requires some stream key from “Download file” which I do not have…
This looks like webMethods.io, not Integration Server. I don’t have experience with webMethods.io to guide on this. Perhaps someone else does.
As an FYI, “webMethods” is a brand name that encompasses many, many products. There is no product that is named only “webMethods”. For the post template item " What product/components do you use and which version/fix level" it is helpful to be specific so that the right guidance can be given.
