Splitting a DocumentList

, ,
1

Hello All,

I would like to know if there is a way by I can split a documentlist into 2 or more document list without having go through a loop.

Any help would be much appreciated.

Regards,
byspeaks.

2

That depends entirely on what is used to determine which items in the original list go into the target lists.

What’s the concern with using a loop?

3

Raemon,

Thanks for the reply. The idea is this.

I receive a document list of about 1500 to 2000 (this can scale up) documents per minute. I need to insert this into the database. If I could split this into 2 or 3 documentslist then I could do a parallel processing and insert this into the database parallely.

The concern with using a loop for splitting is there are going to be 2 loop one after another. One for splitting the document and mapping it to the target document to create a new document and the other is to insert it into the database.If one of these could be curtalied than the performance is suerly bound to increase.

I would really appreciate if you could tell me an alternate way to achieve this.

Regards,
byspeaks

4

There are lots of ways.

  • Ask whomever is sending you 1500-2000 documents at one time to send them one at a time. Or in smaller batches. How big are the documents? Getting a list like this to process all at once can be problematic depending on the document sizes.

  • Loop over the list just once. Transform and store. No need to split that into two loops.

  • Loop over the list and create thread for each entry using doThreadInvoke. Be careful about this one since you can easily flood your system with too many threads.

  • Break up the list using using a Java service into groups of 50. Spawn a thread for each group.

  • Are you using TN? If so, you can use it to help you with parallel processing and throttling things so you don’t kill the server.

These were just a few options off the top of my head.

5

Thanks . That was helpful . I guess I would follow 4th suggestion.