For large lists, appendToDocumentList performs poorly because of the way it is implemented. Every time you call appendToDocumentList, it basically creates a brand new list with size equal to plus 1 (assuming you’re appending one item). It then copies all the items from the original list to the new list and puts the appended item at the end of the new list. This frequent memory reallocation and copying of data is what gives you the performance hit.
When you use an output array, you can assume that the output array is allocated once with the same size as the input array so you don’t run into this problem. The problem with output array is that if you have a condition within the loop (e.g. a BRANCH) that prevents you from mapping a value into the output array, the element at that index of the array will be null, which may not be what you expect. That’s why many folks have indicated that you should use “output array” when the source list and target list are of the same size.
I ran a test for a customer a few years ago to compare different methods of mapping a source list to a target list involving large lists (up to 100,000 items), and at the time, they ranked as follows from fastest to slowest:
- Java Loop: looping done purely in Java with a Java service
- Implicit Loop: for simple lists of the same size, you may want to link them directly in a MAP step and let the IS handle looping implicitly
- Explicit Loop: using a LOOP step and its Output Array
- Append to Array List: similar to append to document list except that an array list is used in the background so there’s no reallocation and copying of data. It is important to set an appropriate initial size to maximize performance.
- Append to Document List: using the WmPublic service
- Dynamic Index: using a LOOP step without specifying Output Array and mapping to a specific item in the output list using a variable index
NOTE: for methods 1 through 4, the time taken to copy the lists grew linearly as the size of the list grew. Whereas for methods 5 and 6, it grew exponentially.
Hope this helps,