List construction takes exponential time?

, ,
1

As our data sizes increase, we’re starting to notice process time climbing exponentially, and my main suspect is document list construction.

We construct document lists by iterating through some incoming data and calling
pub.list:appendToDocumentList. Now, from observing the Java API, it would appear that WM stores lists as Java arrays. If this is so, this would imply that each time we append a document to a list it allocates a new array, and copies the contents of the old array into it. That’s fine when we have a mere 100 or so items, but by the time we build a list of 1800 documents the memory allocation and GC overhead is pretty bad – and would explain our observations.

So: does WM implement lists as arrays that are rellocated when appending new items? If this IS the case, we can work around it, but it would be nice to know that we’re on the right track.

2

We had noticed similar performance issues especially in the case of multiple document list construction inside a single flow service. We created java service and used the server APIs to build the document list instead of using the built-in service and improved the performance by more than 3 times.

You could explore the same if interested.

3

As one wM specialist once told me, it is as you suspect. pub.list:appendToDocumentList reallocates the array when adding new documents.

Berny

4

Since DocumentLists are Java Array types (IData), pub.list:appendToDocumentList cannot use a list. It has to reallocate the array every call, since it needs to have a legal IData in the pipeline each time it is done. As you have seen, this falls over after not too many calls.

Unfortunately this is not described in the Build In Services Guide, but it is described in the GEAR performance tuning docs. See the WmSamples sample.complexMapping.largeDoc for true List services that you can use to avoid any re-allocation.

You need to:

  1. add a call to createList to the empty List before you loop
  2. replace the pub.list:appendToDocumentList to addToList
  3. add a call to listToArray after your loop
HTH, Fred