We have a requirement wherein wM AT would retrieve a list of csv files through SFTP from a single directory and these files need to be merged into a single file before being SFTP’ed to target server.
I have looked at the task actions available in AT and can’t seem to find a way to do it. Is there a way to accomplish this just using webMethods ActiveTransfer without involving an Integration server flow service / java service?
Each CSV file is hardly 10KB in size and we expect a maximum of 4-5 to be present. Looking to do it the ActiveTransfer way as this is a one off requirement and not something that’s going to be used frequently.
For your explained scenario could think of 2 ways of trying to achieve the solution.
Using Write Content option from Execute File Operation Action
For this you would need to Create 2 MFT Events. The 1st Event will have Find Action to get the CSVs from SFTP and then Write Content Action to write the content from all files to a new File in your system. 2nd event will Find this new File and then either Copy or Move this new File to the Target SFTP. Refer Write Content Task Configuration in the MFT Guide
Using Execute Script Action
Create an Event and first do Find and then Execute Script followed by Copy or Move Action. Need to create a Shell Script which will be used in the Execute Script Action which will do the merging of content from all the Find files into a single file which you can then transfer. Refer Execute Script Task Configuration in the MFT Guide
With regards to Option 1 - Write content task - This seems to be an option to wrote only the meta data related to the files that is being worked on and not the content of the file. I have looked into the File reference variables and General variables (given at the end of the MFT guide) which cold hold the data that can be written into the target file and couldn’t find anything as such. Please can you point me to any section if you feel I’m looking in the wrong direction?
Option 2 - Yes this can be considered. But I’m trying to keep it within the MFT itself but may end up with this if nothing else works out.
please keep in mind, that most of the options will only work when all CSV files being merged have the same structure. In this case you will have to scan the result file and remove the additional column header lines.
Usually, CSV fiiles, which are having different structures should not be merged together into one file to avoid data corruption.