A way to keep the pipeline free of unwanted variables is to call services with a transformer in a MAP step instead of using INVOKE. The input and output variables of the invoked service are not retained in the pipeline. Any service that you can INVOKE can be used as a transformer.
I dont know if its a wise thing to do during a resource intense operation,specially inside loops
The Transformers slows down the process.
How so? Is an INVOKE somehow more streamlined than calling a service via a transformer? I suppose I should do some benchmarking to find out.
As with most guidelines, there are times when they should not be followed. Perhaps this is one of them.
I’ve done some testing to see if transformers are slower than invoke.
The service I constructed looks something like this:
-
Create a string list via a Java service. The service takes a count as input and outputs a list of strings with entry 0 intialized to “0”, entry 1 initialized to “1”, etc.
-
Snapshot the current time. I have a wrapper service for java.util.Date.getTimeMillis().
-
Loop over the list created in step 1.
-
Within the loop, use a transformer to call pub.string:concat. inString1 is set to “string” and inString2 is mapped from $iteration. value is mapped to outString.
-
Snapshot the current time.
-
Compute the difference of current time from start time and place in transformerElapsed.
-
Drop interim variables to start with clean pipeline. Keep original list.
-
Snapshot the current time.
-
Loop over the list created in step 1.
-
Within the loop, invoke pub.string:concat. inString1 is set to “string” and inString2 is mapped from $iteration. value is mapped to outString.
-
Snapshot the current time.
-
Compute the difference of current time from start time and place in invokeElapsed.
This service was run ten times for each source list sizes of 10, 100, 1000, and 10000.
List size (# of loops) Avg diff per list entry
10 6.12 (milliseconds)
100 4.679
1000 0.5611
10000 0.20421
Avg diff per entry in millis is the average difference in milliseconds for each entry in the list, with invoke consistently performing faster. The difference between invoke vs. transformer for processing all 10000 entries averaged 2 seconds.
IMO, this difference isn’t significant, even for a small loop. I can share my service and results with anyone that would like to take a look. I may be doing something within the service that skews the results, so another set of eyes is welcome.
B2B Server: 4.0.1
Build Number: 697
Java Version: 1.3.0 (46.0)
Java Vendor: IBM Corporation
OS: Windows NT 4.0 SP5
Dell Optiplex GX110, P3, 733, 512MB
B2B Integrator and Server run on same machine.
we were told conflicting information about the proformance difference
I agree that there seems to be some conflicts regarding exactly how the transformations work.
On the plus side, I like the fact that they keep the pipeline all clean. It also forces lazy developers to declare input/output.
On the downside, it is more difficult to debug when an issue arises.
Last week, I read through all the documentation again to try and figure out ways to optimize our coding and the issues of the use of transformers just glared out at me. So, I used them to produce the raw output of an EDI 850 outbound. I made a couple of small mistakes that took much longer to debug as a result.
Does anyone have any thing to add regarding debugging while using transformers?
Thanks. Ray
A couple thoughts here:
-
This again shows the need to test different constructs to see which performs better for a specific solution in a specific environment.
-
As with all guidelines, the original guideline was for a specific objective/principle. Performance was not a consideration. I would caution anyone against using transformers assuming that they are faster than invokes. You need to test for yourself. I know it’s difficult to profile B2B services to see where bottlenecks are but the same time-proven approach to performance applies to developing B2B services–develop the “right” solution first, then profile/benchmark to see where bottlenecks are (if any) and make adjustments that strike the balance between improved performance and retaining maintainability.
My .02
Rob
What was more difficult about debugging? To step through the transformers, you step into the map step that contains them. You can then step through each transformer, seeing the result of each.
You can also set a breakpoint on a transformer.
Is there another technique that you wanted to use but couldn’t?
Rob
Totally agree on the not making a direct correlation between performance (gain or loss) and using transformers… It is a definitely performance hit if one uses list transformers such as appendToRecordlist when the source and/or target lists are large… The Transformer (I believe?) makes a copy of the pipeline elements to perform the task… Direct invoke of the appendToRecordlist does not. So we see a 30% to 40% increase in Ram consumption when using transformers like the previous example and the source or target list has thousands of elements…
A good case to bench mark processes trying both methods to determine the most efficient way to code a process.
This thread has been ended for some time but thought I’d add in this tidbit of information for those that may come across it.
“…performance hit if one uses list transformers such as appendToRecordlist when the source and/or target lists are large.”
The performance hit in this scenario is two-fold:
-
Before IS 6, the variables passed to the transformer were deep-cloned. Large lists, would therefore be cloned multiple times. For IS 6 and later, the variables are copied by reference and thus the deep-clone performance hit is avoided.
-
appendTo**List performs slowly in any case when the list becomes large. “Large” depends on factors such as CPU, memory and such but my testing showed that performance starts to suffer when the list has a few hundred entries. This is because as the list grows, IS allocates an array+1 then copies all the array entries from the original array to the new array. This can consume quite a bit of time as the list grows.
Hello,
Would you then like to perform your test (or a similar appendTo style) to see new differences or similarities of invoking direct or in use with transforms (map steps). I am not exactly sure the difference between B2b 4.x and IS 6. Maybe this would be better as an Integeration Server tip? Good day.
Yemi Bedu