Large file handling in Broker

, ,
1

Hello Experts,

Could you please explain the process for how to handle the large file handling by using Broker.

Thanks,
Prem.

2

Did you mean by using node iterator for handling large xml or flat file (use iterate=true) and publishing each chunk to broker and then process it?

Can you explain what you mean by “how to handle the large file handling by using Broker.” ?

3

Hi,

Could you please help me in handling large files like 17 GB in webMethods using pub-sub model.
I have read through the bolgs that we can process it using chunks. Can anyone guide me through the steps, or pointers which can help me.

Thanks

4

There are different ways of doing this. If xml files you can use xml node iterator and process them node by node in the file. If flat file you must use ff iterator with ff schema and dictionary created in first place.

The above details can be found from the built-in services and flat file adapter guides.

Also look into the large file handling capabilities provided out of the box with BigMemory Terracotta. Review Integration Server Administration guide.

Last resort is to go with a custom java code which splits the big files into chunks (stream or bytes) and then process these chunks via pub sub.

Altogether you will end up building a large file handling framework which includes staging tables, processing services and reconciliation services if required.

5

Good morning,

I am interested in hearing if this problem was ever resolved and how?

Thanks!

6

My resolution would be a redesign. IMO, pubsub is not well suited to handle messages of several GB in size. I know of no messaging infrastructure capable of handling this. I’d transfer smaller messages with references and pass the net load on some other way (FTP, S3, …).

7

Your point is valid. Broker is a messaging facility and messages should be used for “signal transport”. Not for (sizeable) data transport.

We have implemented a solutions for a customer that tried to send data sets of between 25 and 100 MB in size (and sometimes larger). Their initial implementation was using Broker. I can tell you that that did not work too well.

The final implementation uses DataFeedr to split the file into manageable (and predictable) units of work and pushed them onto the Broker not only safeguarding against killing the Broker system but also allowing them full control over throttle, memory usage, thread pool usage, etc.

If this is still a problem, happy to share the details.

Christian

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.