If we want to do a bulk insertion from the Broker to the target database:
Suppose we have say 500 documents and we published all the 500 documents onto the Broker
and now if there is a condition like:
I have to insert all the 500 documents into the target database at once ( not one by one) or I should not insert even one document.
Also, something like " even if one document goes into the database, all the 499 must go, otherwise not even one document should go into the database"
How would we do this? Is it possible in webMethods, using webM broker to do a bulk insertion into the target database?
Did you ever experience loosing documents on webMethods Broker? If yes, how do you retrieve the lost documents.
Use the JDBC Adapter and explicit transactions. You can start tran, write 500 records, commit or rollback.
Loss of documents virtually always caused by subscriptions being cancelled–which isn’t a loss at all, but correct processing though sometimes “unexpected” processing. “Lost documents” by definition are unrecoverable. Would need to go back to the original source if a document needs to be republished.
Batch insert will guarantee all or nothing. However, If 500 canonical documents are published to the Broker, subscriber needs to get all 500 canonical documents from the Broker before doing any db transactions to make sense. Otherwise, how will transactions help? If we publish all 500 records as one canonical document or a document list of canonical documents, what is the benefit of using Broker and canonical document?
If the architecture dictates to use Broker and canonical document, we can still meet the requirements with some custom coding and workaround solution. I am just trying to weigh the pros and cons between architecture and implementation objectives.
If I was a customer and was told the middleware has guaranteed delivery feature which is one of the selling points, I will not be too happy to resubmit it. I will not be happy about not getting the data to target in time either. If middleware did receive a source, a file in this discussion, recovery should be transparent to customers since we have the source data.
Any thoughts?
TIA,
Mamtha
Basically all the facilities to “group” events together (e.g. appSeqn, appLastSeqn, etc.) have been deprecated. You may be able to use an activation identifier and a JOIN to group docs together.
“If we publish all 500 records as one canonical document or a document list of canonical documents, what is the benefit of using Broker and canonical document?”
If the 500 documents must always be treated by all receivers as a group, then they should be published as 500 items within a single document, not as 500 independent documents. This question usually pops up when people try to publish line items of a PO as independent docs–they are not independent, they are part of a PO and so should part of the PO published document.
If the 500 items are really independent, and some of the subscribers really process them as independent items, then publishing 500 documents is the right thing to do. But if one of the subscribers still needs them as a group, then the target process needs to be designed to collect the 500. You may be able to use the join facilities of subscription triggers to do this. Another approach is to use a staging area to collect docs until they’ve all arrived. Key to any approach is knowing when you’ve received all the documents–that must be taken care of one way or another.
“…what is the benefit of using Broker…”
You probably don’t want to get me started on that topic! There are lots of threads where I’ve voiced my opinion on that–IMO, the vast majority of business integrations do not need it.
“If middleware did receive a source, a file in this discussion, recovery should be transparent to customers since we have the source data.” It’s your job as the solution developer to make sure the solution saves the data somewhere to allow for reprocessing without going back to the source. Trading Networks can be used for this. So can Modeler processes. Even so, sometimes it may be necessary to go back to the source system to have data retransmitted–and the integration tool will always get the blame, regardless of the root cause.
Broker will not hold on to a document once it’s been processed. And if there were no subscribers at the time of the publication, it will not be transmitted to anything nor saved (Broker doesn’t have a “dead letter queue” like some other systems).
Off-topic rant: “…transparent to customers…” Are they really customers? Are they literally paying for a service (not funny money or internal accounting)? Or are they simply in another group using integration services? If the latter, would strongly encourage not calling them customers. A much better relationships between internal groups is as partners delivering goods and services to the real customer that buys these goods/services from the company. Referring to internal groups as customers often sets up expectations that cause confusion and frustration for everyone.
There are lots of threads where I’ve voiced my opinion on that–IMO, the vast majority of business integrations do not need it.