Currently, our solution has two workflows called AddOrderCreateTransportShipment and ReceiveASN(modelled in webMethods BI and implemented in webMethods EI) implemented with an adapter called TDM Adapter
The AddOrderCreateTransportShipmentworkflow will perform the following business logic
Subscribe to document type addOrder
Upon receiving addOrder, the workflow will perform
the Operation (LockOrder)
Upon receiving addOrderCreateTransportShipment
document, the workflow will perform the Operation
(Persist Order)
Publish Document postPersistOrder
Upon receiving postPersistOrder, the workflow will
perform the Operation (MapToShipmentPlanning)
Publish Document ShipmentPlanningRequest
Another Adapter (TransAdapter) will subscribe to the
document ShipmentPlanningRequest, create the
corresponding shipments in the Transportation
Planning System and publish the document:
postShipmentPlanningRequest
Upon receiving postShipmentPlanningRequest, the
workflow will perform Operation (PersistShipments)
Publish postPersistShipments
Upon receiving document postPersistShipments, the
workflow will perform the Operation (UnlockOrder)
The ReceiveASN workflow will essentially
Subscribe to document type ReceiveASN
Perform the Operation (Lock Order)
Perform the Operation (UpdateOrderFromASNInfo)
Perform the Operation (UnlockOrder)
Note 1:
since webMethods BI is used to model the workflows and generate the implementation in webMethods Enterprise, all the above operations are generated as individual Integration Components in webMethods Enterprise
Issues
During our stress test environment, we found the following deadlock and performance issues
DeadLock
TDM Adapter receives an addOrder message, then the
adapter call integration components LockOrder to lock
the order then publish the document
AddOrderCreateTransportShipment. Meanwhile, a ReceiveASN message (refer to the same Order) comes in, the TDM Adapter Thread switch to process the ReceiveASN message, try to ‘Wait’ to lock the order and holding the thread accordingly (until timeout). In a stressed environement, this will result in a deadlock since all TDM Thread (Receive ASN) may wait for a order to lock while no thread are available to continue the AddOrderCreateTransportShipment workflow to complete the processing and unlock the order
2. Performance Issues
There is no document priority (???) in webMethods Enterprise Server.
Assume there is 200 addOrder message in the TDM Adapter Queue. The TDM Adapter will process the 1st message, publish the addOrderCreateTransportShipment. This document will then be re-queued at the end of the TDM Adapter queue and have to wait for the other 199 addOrder message get processed before its turn.
This result in really poor perceived performance since the TDM Adapter is not trying to finish a single flow first. Thus all addOrder get processed, but none get finished. This also causes unnecessary timeout on ReceiveASN workflow as well
Would appreciate if anyone can shed some light on the possible solution here ??
We have try using 3 TDM Adapter instead of 1. 1 Adapter for the LockOrder, 1 Adapter for other processing and 1 Adapter for the Unlock Order. This will eliminate the DeadLock Issues. However, this does not
Just throwing out some possibilities here. Hopefully one of them will be helpful.
Define/use another adapter for the ASN messages.
Don’t publish a message if the operation is just going to come right back to the same adapter. Invoke an operation instead. Or shred this processing out to another adapter.
How is the locking being done? I ask because the answer will determine the possible options.
Publishing a message/document/event is inherently asynchronous and inherently sends processing to the “back of the line.” It appears that steps 1-7 should be straight processing, with no publishing (unless you want to notify other subscribers of activity–but the TDM adapter wouldn’t subscribe to addOrderCreateTransportShipment nor postPersistOrder). Don’t publish an event unless you want another process/system to handle the event.
The ShipmentPlanningRequest should probably be a request/reply operation (publishRequestAndWait). Upon receiving postShipmentPlanningRequest, postPersistShipments would not be published (except as a notification if desired) but would be processed by calling operations.
As I’m putting this reply together, it strikes me that BI has possibly led you down a path that isn’t quite right. Designing a workflow assumes that various steps are being performed by different components. In this case it seems you have one component doing the bulk of the work. Thus, no progress is made because everything is queued up for one adapter, waiting its turn.
Eliminate some of the publishing steps and just invoke operations to do work that doesn’t need to be asynch. If you want things to be asynch then define separate adapters to process each of the docs that are published. That way every step can make progress.
Of course, this opinion is based solely on what is posted here and may be WAY off base.
I agree with what you said. The use of BI (where another team made the decision and I just inherit the solutions) leads to these issues by assuming async integration. BTW, is there any way to ‘force’ BI to use one Integration Components ???
We are exploring the use of multiple Adapter instance here.
The Locking and UnLocking are essentially a simple DataBase operation, where the lock is inserting an entry into a database table, and Unlocking is deleting ‘that’ entry from the database table.
I think re-writing the integration using one single integration component are the way to go anyways. But, as I said, EI does now work well for a complicated workflow though ???
Based on your comments in my other questions regarding ES vs IS and assume that I have to re-do the integration workflow anyways.
Not familiar with IS / Developer (Flow Service) very well.
Do you recommend that our team go ahead / should consider doing it in IS instead of re-do it in webMethods Brokers.
Any experience / inputs (such as Pros/Cons, high-level decision criterias, time considerations) welcomes Hopefully, it does not become a consulting projects. High-Level ideas are good enough for me to pursue the future direction.
Note: The configured operations that we have are mainly EJB calls and simple database (insert/delete/update) operations.
wrt forcing BI to use one Integration Component–I don’t know. Perhaps someone else can chime in?
“…exploring the use of multiple Adapter instance here.” Keep in mind that it isn’t just multiple adapter instances, but adapters with different roles, subscriptions, etc. I think that’s what you meant but thought I’d be sure.
With the use of a table entry for locking, that means you can “explode” your solution out to multiple adapters. As mentioned before, you could have one adapter for each of the doc types that are published, all potentially interacting with the db as needed for processing.
EI does okay for complicated processing. The key is to structure the configured/scripted operations such that they can be easily called. I think that EI can certainly handle steps 1-7 and then 9-11 with no trouble at all.
Indeed, you are right and I mean Adapters with different roles and subscription.
Regarding EI for handling ‘complicated’ workflow, actually, what I mean is in our workflow, we have a lot of ‘error’ situation where we want to stop processing (post an document with error coded embeded in the document)
once error exists. We have used to highle -nested ‘BRANCH’ operation to do some of it (i.e If no Error, do this, else).
Is there an easily way in EI to say (On Error, then do this, or a 'On Error, Goto)
Hopefully, it does not become a consulting projects. High-Level ideas are good enough for me to pursue the future direction.