As per the webMethods documentation - when encountering large documents, the JMS interface to the Broker provides file streaming involving chunking the message.
What I need is something similar when using the Java APIs to the Broker as we have a requirement to handle large documents and there is a danger they may swamp our ESB. Therefore are there any webMethods Java APIs I can use to chuck up the data but guarantee it will arrive at the receiving end reconstituted and with the chucks in the correct order? Currently I exploit the reflection offered by the ‘toObject()’ and ‘fromObject{}’ methods of webMethods Class BrokerEvent as my code is generic Middleware exploited by a number of Projects but ‘toObject()’ and ‘fromObject{}’ read and write Documents to the Broker ‘as is’.
I can engineer my own solution by using Java zip facilities to compress the data that’s written to the Broker and decompress at the receiving end but compress is limited to a certain percentage of the original size and in the worst scenario only turns a “very large document” into a “large document”. Also in my system, the receiving end which gets the documents from the Broker can run multi-threaded across multiple contains so if I chucked up the data when publishing it would be near impossible to reconstituted the data in the right order at the receiving end. Hence my request to see what webMethods offers “out of the box”.
I don’t have any JMS/Broker pointers for you, but perhaps another approach is workable?
Instead of publishing the large file through the Broker, publish a document that references the file on shared storage somewhere. The receiving side would get the published document and determine from that what file to go get and process in a memory-friendly way.
Thanks for your reply. I have explored the idea of putting the document onto an Oracle database. In fact my starting point is actually a Java class which I currently then convert into a webMethods document via the fromObject() method of BrokerEvent. Therefore I looked at put the Java class on the database then send a special document through saying where it is so all the various subscribes could go get it. So far so good. Unfortunately our site standard requires certain red tape information on all documents published to the ESB including a date and in Java terms this maps onto webMethods class BrokerDate. Unfortunately BrokerDate is not Serializable which stops me putting the Java class onto the database. I have raised a Service Request on webMethods ad I think it’s wrong that BrokerDate can’t be serialized when you consider that the bulk standard Java Date class can be. They are looking into this.
It also stops me compressing the document as compression works on the byte representation of the object and hence Serialization !!!.
Interesting discussion…
In general, what are your thoughts on pushing large files through Broker? Even if we get, let say, one or two a day for a particular message stream.
I kind of like the idea that we store the large file outside the ESP and deliver/publish the notification message on to broker with the location details of the large file.
Currently we are experiencing some big delays when large files are pushed on to the broker. I have landed the job to review the code and see how we could improve.
I guess, my question is:
What sort of timing should we expect when pushing large files (20-30Mb) on to the broker?
Is it a good idea to push large files through broker
The primary concept behind Broker is “near real-time integration.” As such, the expectation is that Broker will push around lots of little documents (send the info as soon as it is created/changes), not a few large ones (classic batch processing). The upper bound of what is considered “large” keeps increasing, but the general notion of avoiding large batch files remains.
The question becomes (as you ask), is publishing large files a good idea? That depends on the specific situation, but IME, the usual answer is no.
I’ve been dealing with sending large files through the broker. There is some code written somewhere in the forums that allow you to split the file into smaller chunks. If you do that, you can publish each chunk as a doc to the broker and then on the receiving side, you piece together the chunks of the file.
A fine approach but one must consider the added complexity. What happens if one of the parts doesn’t make it? How are errors dealt with? IMO, decomposing and recomposing a large file via published documents would be a solution of last resort. Doable but not preferrable.
We are currently using this infrastructure out at a large retailer in Australia. They apparently have the largest webMethods infrastructure in the world. Over 1500 stores all running webMethods IS, communicating through brokers to a central system.
I get your point of a chunk not being transmitted or failing. The way we solved this is to receive an ack back for every chunk sent and use the repo to verify that all chunks are delivered. Another thing is that all the stores are on the same network, so loss in data is very minimum.
