What is the best way to process batch data as notifications

Hello Experts,

I have some thing to discuss about the batch processing in DB notification.

A Basic notifications is configured for Oracle Intelligent adapter with Maximum output is 1 record and the component is simple to just receive the input, map it to document and publish. And here the questions.

  1. Is there any way to increase the performance if the output is restricted to 1 record in a notification?. Takes more than an hour to process 1000 records.

  2. The output number of records is increased in notification, there is a possibility of duplication, in between the adapter/broker is down as the transaction would be rolled back and start from scratch in next polling. Is there any way to avoid duplication?.

  3. Is the adapter supports parallel processing to process the batch faster?.

  4. Since threads is not supported except ILA, what is the feasibility to use the # Adapter Process parameter in adapter configuration?.

  5. Is there any other reliable way to Guarantee once and one delivery and increase the performance also?.

Let me know your thoughts.

=Jeya=

Let me start by saying that the ES support event driven, asynchronous, near-real-time processes. Batch processing is typically a time/user-triggered, synchronous process.

In other words it is desireable to avoid implementing a batch process using the ES. For practical reasons this can often not be avoided.

My first advice is to find out if it is at all possible to avoid the batch process. I assume that the information is inserted into the database using a batch process which is why you all of a sudden have a 1000 records to process. This is probably out of your control, but it is worth checking as it will improve the process and avoid your current problem.

In answer to 1.
There are many ways to improve performance, depending on what your current implementation is. You need to provide a lot more information. How many adapters and what types of adapters are you using? What is the process? What is the distribution of adapters on the network relative to the broker? How are your processes implemented, java, flow, blueprints, integration components…? when and How are you using publish/subscribe and request/reply? How many document types are you using? Are the documents volatile, persistent, or guaranteed?

In answer to 2.
When you try to marry the concept of a database transaction with an asynchronous process then you will have to implement your own cross system transaction management. There is a reason why noone has done it. The solution is to break the data down into the individual pieces of data and process them independently, which is exactly what the single notification of the db adapter supports. The best option you have is to write all your adapter processes in such a way that they can handle receiving duplicate information, for duplicate information you will get, regardless. NEVER assume your adapters never will receive duplicate information, because they will.

In answer to 3.
What do you mean by parallel processing? Multiple adapters each polling the database and publishing? If this is what you mean, then I don’t think it is supported.

In answer to 4.
The number of adapters will process the batch faster, but it introduces another potential problem. The events/documents runs the risk of being published out of order, which may or may not matter to you. This depends on the processing logic in your adapter.

In answer to 5.
Use a Queue implementation. Put all your records on a queue rather than in a database and set up many adapters to poll the queue. This is essentially what happens when configuring the # of adapters. A master adapter acts as a queue manager for a bunch of slave adapters. The same problem of order is introduced.

Hope this helps,
Andreas

In asnwer to 4.

how can we handle the ampersand ‘&’ character in a xml document when it arrives to TN ? Can this be parsed thru a given document type?

The XML document should be properly encoded by whomever is generating it. This means the characters like ‘&’, ‘>’, ‘<’, ‘"’ and ‘’’ should be escaped–meaning converted to the standard character entities: &, >, <, " and '

The standard tools used to generate XML (DOM, SAX) normally do this automatically. If the XML is being generated directly by some custom code (often the case) then that code needs to perform these substitutions on the data. Without these substitutions there is NO XML parser that will be able to successfully parse the document. IS will automatically convert the character entities back to their proper characters when a properly escaped XML doc is received.