The following type of Flow code is very common, but I have hit a significant performance issue with it:
Doing some performance testing, I found that if the document is large (say 20Mb) with many fields, then this test can take 200-300 milliseconds! Clearly webMethods is not just checking for the document’s presence in the pipeline - I assume it is converting the entire document to a string representation prior to carrying out the test, which would explain why tests like the following work:
The same performance hit occurs for evaluated labels e.g.
%theDocument% == $null.
While the Service Development guide does state the variable being tested must be a String or “constrained” Object, it seems surprising that WM can’t efficiently handle a simple null test for a document.
I am aware of couple of workarounds:
- Instead of checking whether the document exists, check for a document field we are confident will have a value if the document exists. This is fast.
- Call a Java service that checks for the document’s presence in the pipeline (bizarrely yes this would be faster if workaround#1 can’t be relied on).
Has anyone else encountered this behavior? How did you resolve it?
Have you tested this case?
BRANCH on '/theDocument/someField'
>> $null: SEQUENCE
Yes, that’s my workaround #1. I have found it to be hugely faster. Failing anything else I will look into the application to see whether the documents being tested have a field that is guaranteed to be populated whenever the document exists, and change to this type of test.
Created a new request on Brainstorm: 08226 “new BRANCH labels”
Did you try:
BRANCH evaluate labels true
This only tests if the variable exists.
Hi @Sander_Brinkhuis, thanks for your suggestion.
@Michael_Lemaire, does this approach has the same performance problems you saw before?
Best regards and Keep Safe!
Sander, yes that’s the solution - thank you! That drops the test time down to 0 milliseconds (I’m only measuring to millisecond precision for the moment).
It’s interesting that the moment the expression contains anything else at all e.g.
%theDocument% == $null, WM reverts to taking 200-300mS. So the solution to test the opposite condition (where the document does not exist) is fastest when implemented as a default clause on what you proposed, i.e.
@Michael_Lemaire, do you want to share your test code?
Have you created a test set or is it automatically generated?
A second question: does it make any difference if the document does not exist (e.g. dropped) vs
$null (e.g. set to nothing) in this test you just showed?
Nice, I would use
BRANCH evaluate labels true
Saves a branch.
Thanks for reporting this issue. I am part of the Integration Server R&D team.
I am trying to reproduce this performance issue. Can you share a test package to repro this issue?
I probably don’t have a large enough document in my own example, but I haven’t been able to see the same performance results. I would love to see exactly what you are doing since it sounds very reproducible.
Sorry I can’t send you a test package, but here’s a clip of the relevant flow. I ran a series of flow test steps; each step is bracketed by a
~.processTimer:start service and a
~.processTimer:end service. The “start” service calls
pub.date:currentNanoTime and the “end” service calls
pub.date:elapsedNanoTime, and then the elapsed time is written to the server log (along with a description so I know which test I’m looking at). The main challenge is to generate large enough test documents; in my case I have a document tree with perhaps a million fields that generates about 23Mb of JSON.
I ran the tests on a WM10.3 integration server on a (VM) Intel Xeon E5-2699 v4 @ 2.20GHz with 16Gb RAM, Windows Server 2016 Datacenter,
Hi Gerardo, sorry I can’t share the code. The document is live production data so I have been saved the effort of generating it myself The performance test itself pretty simple - please see my response to Camilo. Yes the delay only occurs if the document exists.