Group files with same date/time stamp to Windows directory

, ,
1

Currently, I have a package that receives a file from our SAP system and then takes certain fields from that SAP file and then creates a separate flat file from it. I then look for the current system date/time. The flat file is then placed (not ftp’d) onto a Windows fileshare directory and renamed with the current system date/time as part of the filename. A separate ftp system then comes to that directory to pick up the file and send the flat file to the customer. Yesterday, there was a massive update done in the SAP system, which sent to me many, many files, which resulted in me writing to the Windows directory many, many INDIVIDUAL files renamed with the same system date/time. My package processed them fine, but the influx of files choked the separate ftp system.

Issue: Currently, the package is pretty much set up to evaluate a SAP file to see if the contents should be sent to the customer, if approved, run the SAP file through the package, write the resulting flat file (not ftp) to the directory, and then start over and evaluate the next individual SAP file in line, and on and on. I have to somehow come up with a way to group multiple files that come to me within the same time period as one big file. Append files with the same date/time stamp together. See below:

INSTEAD of me writing to the Windows directory this:

(files are renamed with SAP followed by the unique SAP file (IDOC) number, followed by the system date/time stamp, followed by a .rdy extension)

SAP_4444444_200604271000.rdy
SAP_5555555_200604271000.rdy
SAP_6666666_200604271000.rdy
SAP_7777777_200605311000.rdy

I need to have:

BIGSAPFile_200604271000.grp (first three files appended to each other)
BIGSAPFile_200605311000.grp (4th file being its own separate file because date/time stamp is different)

SAP_4444444_200604271000.rdy (Holding on to orig filename for reference purpose)
SAP_5555555_200604271000.rdy(Holding on to orig filename for reference purpose)
SAP_6666666_200604271000.rdy(Holding on to orig filename for reference purpose)
SAP_7777777_200605311000.rdy(Holding on to orig filename for reference purpose)

After all files are written to the directory, can I perhaps tell my package to run a Windows batch program to group all individual files with the same date/time extension together? Is that a good idea or can you suggest something else for me to place in my code to handle this?

Thank you very much for helping me with this.

2

Hello,
The question is, does it matter based on the contents of the file which ones are grouped together? If not, then you should just set a serialized key generator (like getting a time stamp inside a WM flow) and tag the files like that. Then you can just run a separate service to take a folders worth of files into time groups like you have above, breaking on every minute.

Doing the time stamp early will give you the ablility to better make the incoming data save in an ordered fashion. Good day.

Yemi Bedu