We have a requirement of pulling close to 500,000 records(on an average) per day from the DB and transforming it and then writing it to a flat file and SFTP the file. We tried the approach of pulling around 1000 records at one go and iterating the process until the row count is 0. But the DB team is not comfortable with us making so many calls to their system.
Is there a process that we could use which would reduce the load on both the DB and webMethods and helps us to accomplish the task ?
webMethods version :9.6
Please provide your valuable suggestions.