My apologies if this topic was posted already. I wanted to find out what the pros and cons were for invoking a service directly or using a MAP statement and calling the service as a transformer. Using map statements would allow me to combine several invokes into one map but I’m not sure about any performance impacts. I remember hearing that one was better than the other but I dont remember the specifics, could someone enlighten me??? Thanks!!!
Jim,
The performance would be greater because you are running under multiple calls using a single flow thread.
WM didn’t spend a lot of time discussing debugging because you could probably write a book about how to debug transformers.
I use transformers to introduce new values like sysDate or to transform values (several at a time.)
I try not to run ANYTHING as a transform where I want to catch a specific application level error.
Hope this helps.
Ray
I was told by a wm consultant not to use transformers in a map step because the each transformer makes a complete copy of the pipeline.
Will,
You may be correct in your statement. I have researched this item this morning by going to the current developer guide and all the way back to the early editions of the developer’s documentation that I have.
Here’s a few key items to note (of course, it doesn’t exactly state the conditions as you have):
-
Transformers in a pipeline operate on the same set of pipeline data.
-
You cannot use the output of one transformer for the input of another (DOH!).
-
Transformers in a map step are independent of each other and do not execute in a specific order (sure…). When inserting transformers, assume that the server concurrently executes the transformers at run time.
#3 corresponds with what I have been told several times by WM personnel. The transformers are child threads of the map step and run under the thread control of the map step. When the transformers are all complete and all OUTPUT data for the map has been produced, then the map is finished and the flow code moves to the next invoke step.
Perform a few experiments, like inserting sizeOfList without and specifics, and try to catch some errors.
Ray
There is a thread on Advantage that might be of interest.
[url=“http://advantage.webmethods.com/cgi-bin/advantage/main.jsp?w=0&s=3217208060&targChanId=-536881000&gid=536880004&mid=1610612982&action=viewMessage”]http://advantage.webmethods.com/cgi-bin/advantage/main.jsp?w=0&s=3217208060&targChanId=-536881000&gid=536880004&mid=1610612982&action=viewMessage[/url]
Just to chime in –
While I find transformers to be extremely handy, I rarely use them. It is very easy to be fooled into thinking that fewer Flow step equates to faster, more efficient code because it appears that fewer services are invoked. As discussed above, this is not true.
It is my personal style to only use transformers when introducing configuration parameters into the pipeline at the beginning of a parent Flow. In all other cases, I create an extra step. I do this for a number of reasons but the most obvious one is that debugging is much, much simpler.
Here’s the real deal on transformers:
Transformers do not operate on the same pipeline as regular invokes. They each get their own mini-pipeline that contains only their own inputs. If mapped in, they are “deep clones” of the mapped inputs.
Transformers will generally execute measurably faster that regular invokes. However, if a mapped input is very large (xml node, nested record, etc) performance will be measurably slower due to the cloning plus additional memory requirement.
Good transfromers:
pub.math:addFloats, pub.string:append, etc
Bad tranformers:
pub.schema:validate, pub.web:documentToRecord, etc.
I personally don’t see any difference between debugging transformers and regular invokes. If the transformer is a flow service you can step into just like regular invokes. You can even set a breakpoint inside a transformer and use it with Trace Into.
More than meets the eye.
YMMV
Regards,
Theo
Transformers may not work always as you expected if you use it in the map step with in a loop. So it is not advisable to use transformers within a loop.
Good point, Jesin.
Here is another huge problem with transformers…If the service you invoke has the potential of failing (which ALL services do), then the getLastError service will return a lastError record whose pipeline variable only contains the inputs to the transformer…not the full pipeline at the error point…Makes it very difficult to do any amount of error handling if you use flow error handling and getLastError service…
This bug has been reported to webMethods but no estimated fix date yet…
Jesin: can you share any specific situations where transformers do not behave as expected?
Tony - this is due to the transformer using it’s own mini-pipeline. Your transformer should do it’s own error handling.
I don’t know if I would consider this a bug or just a limitation due to the way transformers work.
Theo
Just a note for anyone researching transformer performance and coming upon this thread…
Integration Server 6.x changed the behavior of transformers and the interaction with the pipeline. According to information in “GEAR_6_Performance_Tuning_White_Paper.pdf” inputs to a transformer are copied by reference from the pipeline. Previously items were cloned and thus had significant performance implications.
In version 6.1 the guideline for when to use transformers to add performance to your flows should be:
Do use transformers:
- for all pub.string, pub.math, and other pub services using string, stringlist, or document list as the inputs and outputs
- for all user created java services that use string, stringlist, or document list as the input and output
Do NOT use transformers:
- to call FTP operations. I had problems with VAN.ftp operations in the WmEDIforTN package when I called them using a transformer.
- to call anything using a node or object as input or output - this led to an error about an dereferenced object.
- to call services that are complex such as mainframe adapter transactions - Mainly because these are confusing enough already
- to call trading networks services? Has anybody successfully called
trading network services like tn.log enclosed in a transformer? Anyway these are all mostly in catch steps and we won’t save much processing time by enclosing them in transformers. - to call flow services that have try/catch logic where you want the catch to propogate back to the originating flow.
- to call any flow that needs a field or returns something in the pipeline that isn’t specifically defined in the input/output (like the wm.tn.doc.recognize flow where the edidata field has to be present but not mapped to an input field).
Does this sound like a “best practice”? What would you add?
Hi Jon,
I tend to limit use of transformers to situations where multiple pipeline variables can all be modified at the same time in a single map step, and it is obvious that there is no significance to the order of modification.
Happily, this tends to correlate well to you suggested “do list”, but I do it for easier understanding of flow code.
I do not create map steps that contain a single transformer for the reason above.
If I can’t get good enough performance in this way, I look to other approaches for optimization.
Hi Jon - To piggy back on Rob’s comment, the part about transformers “passing by reference” as opposed to “passing by value” certainly enhances performance. But your “service in and service out” parameters of the transformer services are not available to the pipeline after calling the transformer.
A quick way to test this and possibly quantify the performance gain is to write 2 flows. One that invokes pub.math:addInts directly and another that calls it via a transformer in a MAP step.
I haven’t done this and would love to see if anyone has done something similar to provide any quantitative analysis.
Cheers,
Rajesh
There are certain WmPublic services that don’t work when in a transformer, but do work when invoked explicitly. For instance, I just encountered this issue with pub.file:stringToFile. It silently fails when in a transformer, but worked flawlessly when I invoked it explicitly.
This should be reported to SAG support. There might be other services which can behave the same. Looks like a bug to me.
Have you tried the same by installing the latest IS core fixes?
If the transformer doesn’t have a output link, it won’t be invoked.
1 Like
I am on the same page. The services invoked via transformer should have a link to Pipeline Out. I assume the behaviour you see in the above is by design (I tried the same on 9.12 with fix IS_9.12_Core_Fix10) however for more details you can contact SAG for more details.