Node Iterator and Large Documents

, ,
1

We are using a node iterator on a large document. The document is over 2GB in size and has about 2.5 million product line items (its a PIP2A1 from a vendor). We iterate over the line items and merely do a documentToRecord on the node returned from getNextNode. On or about the 2.25 millionth line item, the documentToRecord throws

Error: java.lang.ArrayIndexOutOfBoundsException: Array index out of range: -262144

I assume that since getNextNode returned a node, there is not problem with the element itself.

So, I’m wondering if this might be a filesize issue. Any file over 2.1GB would exceed the precision of a Java integer (right?). So, if wm is using an integer to keep track of the file position, this could be a problem, though one would have thought such a problem would surface on the getNextNode, and not the documentToRecord.

Any ideas? Has anyone processed a document greater than 2.1GB?

Thanks

2

Are you passing “criteria” parameter in the NodeIterator service,specifying the 2A1 header record + lineitem record structures in that param?

HTH.

3

For the criteria, I’m specifying “ProductLineItem”. This all works fine for our “smaller” updates, of say, around 1GB. MovingWindow is set to “true”.