Fetching 15k Records Using JDBC Adapter Service

, ,
1

Hi All,

We have requirement where we need to fetch 15K records and insert into other db. I am looking for best way to do this?
one way which I am thinking of is fetching 1000 records at a time and processing it? Please any of let me know if any other alternative best option to do this?

Thanks in Advance,
JSree

2

The best way I think is to.

  1. Select.
  2. process.
  3. map.
  4. Batch insert.

This entirely depends on what kind of processing you are doing, the system capacity and your architecture. Is your IS capable of handling 1000 records, 10000 records or 1000000 records ? If its a straight mapping from one canonical to another without any complex processing and you have followed all the best practices and GEAR doc’s I dont think so selecting and inserting 15K of records using JDBC adapter template should be a problem.

Jiten

3

Be careful with transactions if you are using repeat steps or loops in this data transfer. I would suggest framing and publishing queries for batches of records based on a unique field in the source table. Using trigger properties you can fine tune the performance and control the data transfer speed.

You can first consider the above first approach (All records in one stretch) by Jiten as 15 k records may not be a huge dataset.

Cheers
Guna
http://www.nibodo.com

4

Hi Guna/Jiten,

Thanks for your suggestions. Let me implement this and I will keep you posted.

Thanks,
JSree