So, I have to do some ETL stuff…
I have to move 600,000+ records on daily basis.
I am doing this so far.
Obtain the number of records, then divide by a fixed number (say 20,000) add one and I got the number of times I have to do a dynamic SQL to read the total data.
So I get this number (33). It is the number of calculated times I have to do my sequence of read from the source and then insert on the target.
Now I face with the following scenario:
Got number (single number in a variable)
Loop (33 times)
Select data (chunk of 20,000 records)
insert data (20,000 rows)
The number of records varies day a day, sometimes is less than 40,000 other 500K+ records. So fixed document is not an option.
I am pretty new with IS, but I have worked with several ETL tools.
Thank you in advance for your advice.