Implementing local variables

, ,
1

The use of local variables is crucial to prevent services fouling up the values in the services that called them.

It seems to me that there are two ways of reliably implementing local variables for services in WM6:

  1. Insist that all services are called as transformers in MAP steps, and never use INVOKE steps. This provides automatic scoping of the called service.

  2. Wrap each service in a series of steps that:

    • create a local scope document,
    • assign the input paramters to the scope document,
    • execute the service’s logic in a scoped block.
    • retrieve output from the local scope document and assign it to the service’s output parameters.
    • drop the local scope document.

Am I missing something obvious? Is there a better way?

2

We use the local variable holder pattern:

http://www.customware.net/repository/display/WMPATTERNS/Local+Variable+Holder

No need to execute each service in scope, just clean-up anything that isn’t needed later or else isn’t an output

NOTE: you should only ever be leaving variables on the pipeline that are listed as outputs. That way the design time view will show you that you are going to “foul up” variables.

Simple solution however: put a MAP step at the end of the service that drops anything that’s not supposed to be there… and always map outputs that you’re going to use to a new name.

regards,
Nathan Lee
WmUnit - Improve the quality of your services
http://www.customware.net/wmunit

3

I believe that it is a performance hit if you call each service as a transformer in a map step (when calling just one transformer), and obviosuly, it is poor for readability of the service.

We call pub.flow:cleanPipeline after every service to eliminate any additional values, either explicit or implicit (hidden).

There is another practice (which I do not like) where the inputs are variableIn, and the outputs are variableOut.

If its something of concern, I would also avoid common names, such as “input” or “inString”.

Lastly, you can use the scope property of the service to restrict specific values to the pipeline (ch. 7 of the developer users guide).

4

I agree that the “local variable holder” pattern can alleviate the problem (provided you don’t have any recursion). I think it’s worth pointing out though that those values are still in the pipeline, albeit in a document. It still leaves the (to me) sub-optimal software engineering situation where all variables are global.

I agree too that using transformers results in less readable logic, however to me reliability and black-boxing are higher priorities. I doubt that the performance hit for using a MAP with transformers would exceed that of manually creating a scope document (which I suspect is what WM is doing internally). Back in the old days many programmers didn’t like stack-based local variables as in C, Pascal, Java etc. because the terrible performance overhead of maintaining a stack!

Putting the MAP at the end of the service to drop local values doesn’t always work, I’ve found. Nor does clearPipeline. I spent a lot of yesterday writing some short test services that proved this. For instance, if the calling service-1 has a pipeline document called, say, “vars”, and the called service-2 also uses a document of this name, then on return to service-1, the “vars” document retains values from service-2, even if service-2 called clearPipeline or explicitly dropped the values in a final MAP.

5

The above points are valid, very important and doable within webMethods.
Building reusable “black box” component is a concept that webMethods aim towards which is demonstrated by the ability to take a FLOW service (written once) and have it able to be called by a either a HTTP client or SOAP client.

Some of the issues that you are experiencing and have uncovered via your testing are covered by the webMethods Education produced courses - specifically:

  • how we form the pipeline object,
  • what type of objects are contained in the pipeline,
  • how the pipeline is passed by reference from one step to another,
  • how a pipeline is cloned when we call an inline transformer,
  • how we should optimise our code to create that optimal “black box” component.

Reference Course D127: wM 6 Advance Developer Topics.

6

I think every language must implement the stack model when reusable components are invoked.
In my opinion, if you must write a “black box” flow service do first of all a “pushPipeline” and last of all a “popPipeline”, both methods are easy to write in java.
This way you may even call your flow service recursively.
Remeber that such methods modify your pipeline at runtime, at designtime you may remain confused.

7

Hi again,
I would ask why there is a need for recursion, as I have found very few cases with integration code where use of recursion is even a consideration…

With recursion, IF needed (and I have found I’ve never had to use it in webMethods), I would rely on the use of a temporary scope document as the pipeline to the recursive call.

I’m also not sure how you found that your service left variables on the pipeline if you cleaned them up properly using “drop”. Perhaps you have multiple exit points in your code which is permitting them to stay there?

For the suggestion about cleanPipeline: I’ve found that a dangerous beast from a maintenance point of view as people tend to use it and all too often forget to put values in the “exclude” list. There are much cleaner (I think) ways to leave a clean pipeline by simply dropping variables as you go.

For the training: I’d go the D125 course as well as this introduces the fundamental concepts of the webMethods platform.

regards,
Nathan Lee
WmUnit - Improve the quality of your services
http://www.customware.net/wmunit

8

I’m interested in the tests you’ve done that indicate that Maps and clearPipeline are insufficient to avoid unintentional modification of pipeline variables.

I’ve created and run my own tests using the following three scenarios:

----

Scenario 1:
    Service S1 containing:
        Document D containing:
            String S1 with text "This is String S1"
    S1 invokes Service S2, passing string parameters Input1 and Input2 and expecting output String Output
    S2 does the following:
        Uses a Map Step to create Document D containing:
            String S2 with text "This is String S2"
        Creates String Ouput with text "This is String Output"
        Uses a Map Step to drop all objects except Output

Pipeline results after tracing:
    Document D containing:
        String S1 with text "This is String S1"
        String S2 with text "This is String S2"
    Input1
    Input2
    Output

----

Scenario 2:
    Service S1 containing:
        Document D containing:
            String S1 with text "This is String S1"
    S1 invokes Service S2, passing string parameters Input1 and Input2 and expecting output String Output
    S2 does the following:
        Calls clearPipeline, preserving only Input1 and Input2
        Uses a Map Step to create Document D containing:
            String S2 with text "This is String S2"
        Creates String Ouput with text "This is String Output"
        Calls clearPipeline, preserving only Output

Pipeline results after tracing:
    Document D containing:
        String S1 with text "This is String S1"
    Input1
    Input2
    Output

----

Scenario 3:
    Service S1 containing:
        Document D containing:
            String S1 with text "This is String S1"
    S1 invokes Service S2, passing string parameters Input1 and Input2 and expecting output String Output
    S2 does the following:
        Calls clearPipeline, preserving only Input1 and Input2
        Uses a Map Step to create Document D containing:
            String S2 with text "This is String S2"
        Creates String Ouput with text "This is String Output"
        Uses a Map Step to drop all objects except Output

Pipeline results after tracing:
    Document D containing:
        String S1 with text "This is String S1"
    Input1
    Input2
    Output

----

The conclusion seems clear:
    Upon entry Flow Services should always call clearPipeline, preserving only their inputs.
    Before exit, Flow Services should either:
        Call clearPipeline preserving only their ouputs or
        Use a Map step to drop all objects in the pipeline except for their outputs.

cheers,
Mike

9

Three points:

  1. I guess I should have mentioned that I was responding to mlemaire.
  2. My reply was formatted so much better than the way it actually renered.
  3. I'm using this message to determine whether or not I can use HTML to format my messages.

Too bad there is not “Preview” function on this forum.

10

If you write services which:

  • only return the inputs and outputs specified
  • drop any other variables
  • don’t add/drop/modify the contents of passed in documents (unless that is what you want to to)

You will not have a problem I believe.
Modifying the contents of a passed in document is not something which is typically done, that’s why you’re getting back a “changed” document as it is beyond the control of the top level pipeline, similar to in java if you call a service with a hash table, if the service adds values to it, it’ll have those values in it after the call finishes.

I would highly avoid using clear pipeline as it is a lazy and dangerous practice to get into (and is also not reflected at design time in the Developer tool).

Recursion:
to avoid any problems: create a temp document, put the inputs into it and then use scope when invoking, map the outputs, then make sure you blow the temp document.
I’ve used recursion in this way and it works fine, you just have to write good clean services, and use scoped invoke for the recursive call to make sure it has a nice clean pipeline.

regards,
Nathan

11

mike,

I did a few tests to establish what was going on; I’ll describe the simplest one here.

Service A:

  • MAP: create a folder called “vars”, containing a string “name”, assign the value “Michael”.
  • CALL Service B.

Service B:

  • MAP: creates a folder called “vars” (as you do), containing a string “name”, which it assigns “Fred”, and a string “shoeSize”, which it assigns “9”.
  • CALL clearPipeline, removing EVERYTHING.

Run Service A, then look at the pipeline. It contains the folder “vars”, which contains:

  • name = “Fred”
  • shoeSize = “9”.

So, obviously, clearPipeline hasn’t done what you would expect (or what I’d expect anyway :). This problem seems to occur only when variables are held within folders – which we consistently do to avoid cluttering the pipeline and maintain readability in Developer.

Consequently, as per my original post, I have concluded that the only reliable ways to maintain local scoping are either to use transformer calls, or (my preference) to manually and explicitly wrap every service in a scoped block. Other methods may work too, but I’m pretty settled in my patterns now, and the scoped block method means I don’t have to update pipeline-clearing steps each time I add functionality and variables to the service. Scoping issues have been the cause of most of the bugs I’ve hunted down over the past few months – and they can be real hard to find as the affected services behave differently depending on how you run them. As a result, I believe it’s critical to have a simple, reliable, no-intelligent-thought-required technique to force good scoping.

When we started moving from WM4 to WM6 late last year, I originally assumed that WM6 would provide automatic scoping, given that it takes input/output parameter declarations, but it doesn’t.

I would like to see WM provide an checkbox option on flow services to automatically provide a local scope, and only allow declared parameters in and out of that scope. This should be trivial to implement (it’s exactly what we’re doing in the flow, so it should just slot into the underlying architecture), and would IMNSHO greatly improve application quality.

12

You are absolutely correct. We call the scoped pipeline “pipelets”. Transformers were implemented this improved way. There was a project a couple years ago that included providing pipelets as an option on regular Flow Invoke steps. Unfortunately that project, which created a Flow Engine that could run outside of Integration Server, did not have it’s goodies merged back into the base product. This specific feature is back on the roadmap along with a scoped down set of Flow performance improvements.

Automatic scoping would cause compatibility problems, but we could have the pipelet property default ON for all newly created Flow Invoke steps.

Unfortunately none of my blather will help you this year.

13

I still don’t see why this is a “blocker” kind of issue: the behaviour of the pipeline is a known thing, and if you are really encountering that many problems due to dumping everything in a “vars” document then what you’re doing is essentially shifting the pipeline “up one level” which is the cause of your problems. You need to change your process for writing services to avoid this (just like if you’re constantly writing java code that doesn’t handle nulls you need to start putting in code to check for that sort of thing, rather than relying on the caller to catch random exceptions…)

I’ve found that using the local variable document holder pattern (which is relatively easy to do and doesn’t require messing around with scope) and/or simply cleaning up the pipeline as you go will avoid any problems. Even when people don’t use the local variable pipeline holder, so long as they clean up the pipeline before exiting the service it’s not usually a problem. A simple change of the name of your tidy-up document to something that will be unique for each service that holds the variables will achieve this…

The only time I have really needed to use scope:

  • messy services that are beyond my control to prevent from returning undeclared outputs
  • recursion (though this is just an issue with the hacky private projects I’ve done on webMethods, I’ve never had a reason to use it on a commercial project).
  • when I’ve had to invoke things that return the same named variable as one I’ve got as an input for the current service (and this could be overcome by renaming and renaming back if necessary).

It would perhaps be a useful future feature to have transformer-like invoke of services, but existing functionality would need to be maintained and would need to be able to turn it off, as there are definitely occasions when you need to access things in the pipeline that aren’t inputs… I’d also wonder how you’d get around the problem of mutation of documents, as aside from doing a deep clone of everything in the pipeline, the problem will still occur… And doing a deep clone of every document will be a tad expensive in terms of memory and CPU. In fact imagine the memory requirements of doing a deep clone where you have an entry point service which calls A which calls B and passes in the same document down through each service:

invoke of entry point service ( memory contains 1 x the structure)
– invoke A, passing in the doc ( deep clone of doc = memory contains 2x the structure)
------ invoke B, passing in the doc ( another deep clone of the doc = 3 x the structure)

Of course the actual memory requirements might be a lot lower due to the way strings are held in a common pool in the JVM, but all the other associated structures will be taking up memory, all the IDocs with their internal hash tables and cursors and the like…
Having this feature ON by default would indicate to me that webMethods had struck a deal with the RAM manufacturers :wink:

just a few thoughts, (interesting thread though)
regards,
Nathan

14

Hello all,
One other thing that sounds nice, to me anyway, is to be able to specify a seperate document for the inbound scope as the outbound scope. I regulary take an input document referencee that I would like to scope on and provide a different reference to store my results in.

Oh mlemaire,
Why don’t you call clearpipeline before entering the service? it seems that in not doing so, you allow the upper level document to be referenced and modified, even though you kill the inner-level document.

hey nathan,
deep cloning would not work work for streams, good as large docs tend to work this way. One thing that may be nice would be freezing of the input values unless they are marked for cloninig, it would give a large performance savings. It is a programming feature that I is a available in programming languages, Ruby for example.

Too all,
I would also agree with many that an inbound/outbound local variable document and only one clearpipeline as the first call are needed in flow services to provide strong and clean pipeline handling. I too use a temp document to shuffle copy inputs for recursive calls, and do frown on recursion as it holds well with the procedural nature of webMethods like the Scheme programming language. Good day.

Yemi Bedu

15

Hello again, (Stinking no preview)
Two updates, I don’t frown on recursion and the freezing nature of variables works opposite in Ruby so you can modify it all unless you don’t want to. Good day.

Yemi Bedu