Hi all, long time lurker here, first time poster. I’m trying to fingure out if a concept I have in mind is even possible to accomlpish and I hoped someone here might be able to tell me it really is impossible before I waste too much more time trying to find out how to do it. (Yes I know proving a negative is pretty hard )
The basic premise is this: I have a large batch of work to do, call it a batch of bicycles to put together. I have a flow service that takes a single bike as input and puts it together. What I want to have is some method of looping through my massive order of bicycles and kick off the bicycle creation service many times without waiting on it’s completion. I.e. I want my order for 100 bikes to be processed with all 100 bikes being assembled at the same time.
I found this quote from mcarlson in a thread:
Your question seems to indicate that you want to run two Flow services in parallel each on its own processing thread. You have to write a java service or two to do this. The Service.doThreadedInvoke method allows you to specify a service name to invoke and the IData containing the inputs of that service. See the WmSamples package for one example. You can also search WM Users for doThreadedInvoke to turn up more discussion on this point.
Integration Server is multithreaded. Using multiple threads to perform your integration work is a process of analyzing the work to determine what can be executed in parallel, invoking multiple additional threads to do the work and then collecting and reassembling the results (including any exceptions).
I’m hoping someone can confirm for me that I can write this java service so that it invokes a flow service, but makes a trip back out through my load balancer so that the other servers in my WM farm get a share of the work. Has anyone had any luck doing anything like that?
A Java service and the doThreadInvoke method is indeed one way to accomplish parallel processing, but it won’t get you out to the LB switch. There are a few ways.
You touched on probably one of the easier techniques, depending upon if you need the results of all the bicycle assemblies to come back together or not.
If each bike can be kicked off independently, not needing to join the batch at the end, the easiest way is to do an HTTP get/post to the virtual host/IP of your load-balanced IS instances. The key is to not have the invoked service do all the work before returning a success response–it must receive the document then kick off async processing of the doc (using doThreadInvoke) and return ok right away.
One variation on this is with Trading Networks. Each bike is posted to the virtual host/IP (via the load balancer) to a service that submits the doc to TN. Configure the rule to by async and TN will return control to the service after the doc is successfully saved (and recognized) so that ok can be returned before document processing really starts. This achieves the async without needing to use doThreadInvoke as TN does the async work for you (and tracks it).
Another approach can use the Broker, if you already use it in your environment (I wouldn’t introduce it for only this purpose). For each bike, publish a doc. Configure each IS to connect to the same Broker with the same prefix and have the same triggers and trigger service. Set the configuration of the trigger to concurrent, with max threads and such. Each IS instance will process bikes off the single queue, giving you controlled parallel processing.
Another approach might be via Designer and the process engine (PE). A model can be configure to run things in parallel. It can also be configured to have steps execute on different logical servers (which can be mapped to different physical servers). This would be a different way to use Broker (the PE would publish step transitions).
Hope one of these approaches will work for you or trigger some ideas.
Thank you for responding! This is great information. I wanted to go ahead and thank you in advance though because I don’t know how long it will take me to soak in the firehose of knowledge you may have just directed my way What a nice problem to have!
If I can manage to get this working via one of these solutions (the TN one looks espescially attractive at the moment) I will post back my experiences. Now let’s see if I can grokk what your saying before my deadlines all slip.
Can you elaborate on what exactly you mean by “…when it is just Webservices alone?” Do you mean calling multiple web services in parallel? Or introducing parallel processing within your service that is invoked via a web service interface?
Actually it is the latter leading to the former…Let me explain.
I get a status update from one system via Web Service. This request is read and then 2 separate pay loads(XMLs) are created and then it should be sent to 3 systems for them to update their system, all 3 systems receive this update via Web Services.
So basically
how to do parallel processing by which 3 web Services are invoked without waiting for the other to complete…
for now I dont need to worry about the response…they just send a ack that they got it…however just in case I want to collate them into one and send it back to the calling app…how can I do that
Review the IS Java API. Service.doThreadInvoke() and getIData() for information on creating threads and joining them to get the responses.
Write a Java service that calls doThreadInvoke for each target. The service that doThreadInvoke calls would be an IS service that uses a WS connector to make the call to the external system. Pass it enough details to make the right call.
Call getIData() on each thread returned by the doThreadInvoke calls. That forces a join. Be sure to use timeouts so you don’t wait forever.
Within your service that is invoked via WS, call that Java service. When it returns, all 3 web services should be complete.
You could make this pretty simple by just publishing one doc and have three subscribers. All are updated independently of each other. Unless you are going to roll back transactions or issue compensating transactions then don’t block and wait for things that can be done in a asynch mode. Which it sounds like you are not since you are trying to do this in parallel.
Excellent point. I guess I got a bit myopic on the approach. Don’t even need a Broker for the pub (if a Broker isn’t currently being used)–local publish would work fine too.
)