I am new to large file handling implementation where as my customer has ask us to do POC on large file handling with SAP Idocs to check our expertise on this matter.
So I like to understand how to design and develop processing of inbound/outbound processing of large file of SAP idocs.
System details SAP Adapter version 7.1 and WM 9.7 with OS AIX.
Large idoc with multiple segments greater than 20 K/hours
Questions my customer/sap support team is asking about large file processing on ESB layer as follows :-
What is the max size of sap idocs ESB server can process ?
Once sap idocs is trigger from sap system, How ESB server or How do you determine incoming idoc is large files?
Explain high level business logic you write to handle large idocs file processing?
In Outbound flow While sending idoc to SAP system, Will your ESB send all Idocs at a single thread or it will push 1 by 1 idoc at time in loop ?
If any transient or network break down happen then how/ where do you store incoming large file for time being?
How do you resubmit failed idocs?
If file is large do you ESB server need to twerk memory/extended setting or JVM ?
Can you please share from where do you find these services, looks like these are deprecated ones for me in v7. For handling SAP Idoc there are built in API services found in WmSAP/pub folder.
@ Vinay,
Please check 7-1_SAP_Adapter_Install_and_Users_Guide for more details. We had a IDOCHandler framework custom built in one of my past client projects. The core services used are from WmSAP package.
Since you are using the SAP adapter , I believe you will be receiving the IDOCs via ALE notification. In this case if its a large IDOC and if all the segments are not required, then may be based on your requirement you can select only the required segments in the notification request section tab. Hence only those segments will come to the ESB. In this way you can limit the IDOC size on ESB side. Also if you have TV, you can redirect the IDOCs from notification to TN, there you can do large file handling ( it will be easier I believe ) from there. In our project we used to do that , route the IDOCs to TN , where large file handling settings were there.
So 2 points here.
select only required fields and segments in the notification.
route to TN ( if present ) for large file handline , else save the docs in filespace and use the node iterator logic.
I am about to request to my infrastructure team all SAP fix,version, Jar and lib file going to used for this implementation but still my client/support team like to know about about
Q 2 and Q 5 ie
Once sap idocs is triggered from sap system, How ESB server/middleware or How do you determine incoming idoc file is large files?
If any transient or network breakdown happen in Integration flow then how/where do you store incoming large file for time being? or what is logic or system you use to store this large file so that it should not be lost in flow.
Basically, my client intention is like to know idea here is to resubmit failed data again will cost more to them.
I think it is not possible to detect if an IDoc is a large IDoc or not before it has been completely received.
If IDoc could not be completely retrieved from SAP Server it should be retried.
If this fails the IDoc should remain avialable at SAP side for retry upon the connectivity has been restored.
We are currently testing a new solution with a customer. Once the testing is completed, I will report here.
Our solution works as a companion to webMethods IS and handles those tasks that the Integration Server struggles with, such as ultra-large files. But it leaves the tasks that IS is good at with IS, like it should.
The IS lets the WmSAP adapter store the received IDOCs in the transaction store. Our solution monitors this store and automatically takes over when a file is received. It parses and process the file in an IDOC-by-IDOC fashion (or multiple IDOCs in case desired; a configuration setting). In case an IDOC fails, the error handling deals with that one IDOC separately while the rest continues to be processed. The IDOCs, after processing, are pushed one by one to data handling services (Flow services on IS) over the HTTP stack and handed off to IS for further processing, persisting, or whatever needs to be done with them.
A large file (containing 1,000 IDOCs / 4.1 GB) can be processed on a workstation with limited heap memory (512MB) and 4 cores in 2 minutes. Increasing the infrastructure allows for linear scaling and at the same time the user maintains complete control over memory consumption and CPU thread usage. All through configuration.
We have other customers using this exact same solution for different use cases and different file types but we have found that it also supports massive IDOCs and ALE exports. There are still a lot of companies that dependent on such data and are struggling with it.
If you are interested to hear more, please reach out outside the forum. I am available at christian.schuit@centipod.nl.
