Hi All,
Wanted to know whether sending a data as byte or object to UM queue/Topic is preferable for message size perspective ? I have requirement where large message needs to send to queue.
Regards,
NB
You need to review the config property max buffer size to handle large messages.
The maximum buffer size that the server will accept. Default is 1048576
You can send IData as is, what is your message size?
Hi Mahesh,
Wanted to know if I convert string message into byte or object and then send to UM then will it occupy less memory compare to string ? Will the message size reduce ?
As we know if we convert text file into zip object in our computer ,file size got reduced. Is it the same for UM server ?
I checked by sending string and byte both messages but size was same. I verified the size by checking Event store property of a Topic in Enterprise manager.
My requirement is that message should take less space on UM server if triggers are disabled.
Regards
NB
I am also in a believe that UM can handle larger payloads as Mahesh pointed out there must be some tuning required setting via config param’s and value etc…
HTH,
RMG
Hi rmg,
Yes I agree UM can handle large messages by changing some parameter in config file. More memory needs to be allocated to UM for this.
However question is when we send the data as byte or object to UM it should take less space in UM server as compare to string data.
Like if we convert text file into zip file in our computer then file size got reduced. Does it not the same in case of UM server ?
I would recommend sending bytes over the network.
Kindly explain your use case and the type and size of message payload you would be dealing with.
Hi Mahesh,
size of message will be around 6 mb. Data is in string format.
I need to send the data to UM. SO if target application is down in that case data should be available in UM queue.
No problem, have the required settings on UM and JVM to handle the large file.
Make sure your queue/channel is persisten if subscriber is unavailable or you can look out for durable subscribers. Build a POC around and send both String and Bytes, compare the througput and performance. Let us know if you face any issues.
1 Like
Using UM to move messages of this size might not be the best option. If it was for me, i would use TN or Active Transfer with TN for this requirement. If you have licenses for those products then use them.
Using UM for this use case will not be a scalable solution at Enterprise level. If either the message size or the number of messages increase over period of time then your solution will break with UM unless you keep on allocation both DIsk space and Memory to UM.
1 Like
After subscribing the large message from UM, I am using xml node iterator in my flow service to handle large xml.
After having mapping at the end one document is created and I need to send that to target system. This document is also large and I need to send it in xml string format using pub.client.http service. if i see the size of this document goes around 100mb.
I am worried here if it cause any issue in IS side like java heap memory. Can anyone please suggest on this.
You will definitely have OOM if your JVM does not support such a large file in memory.
Can you tell us what sort of file it is and can this be sent over FTP or using other protocols.
Share us the end-to-end integration flow to suggest you other options.Also look into "Enhanced XML Parsing "
Unfortunately we can’t use other protocols for sending files.
If I use Enhanced xml parsing Moving-window mode can not be used.
Would you like to suggest for increasing JVM. If yes how much it should be. Are there any parameter to decide JVM size.
In IS flow service I see after preparing mapping document that document needs to covert into stri g. And string size goes around 100mb. We need to send the xmlstring to target system that is the requirement and it seems difficult to handle.
Tuning of JVM needs a benchmark testing with the possible max large file content.
Rather sending the raw XML content, did you try sending the zipped format using JAVAs ZIP capabilities.
I second “Akshith Arremreddy” here. It is not recommended to use UM to send across messages of that much size. It is not that UM cannot handle that size ( it is JVM based and any server with fine-tuning can handle), but it is not meant for it. Messaging platform should be used to convey lighter stuff like events for safer and smoother design. Try to revisit the scenario/requirement and see where it fits.
There are many ways to do this,
- Post the large XML to TN and use large file handling techniques to handle the contents in it as individual transactions.
- If you dont have TN license then create a Java service using StAX parser to parse the XML and split it in to small XML files(individual level) and then process them using File poller port.
- Use UM but instead of splitting them by size(6 MB you mentioned earlier), split them by transaction data which should bring down the size of the message down considerably.
I have used StAX to successfully parse XML Files with more than 4 GB for one of the clients on an Integration server which had 4 GB Max memory allocation. I usually split them by number of records per file and then write it to a File share from where a File poller port would pick the file and process the records.
May be you can try send that document as a stream rather than data/string in the http transport.
Plz, Check if this option works on your receiving target side team also.
HTH,
RMG