Best practices for breaking up large CSV files and submitting to Amazon SNS

[Vikash_Sharma1 (Vikash Sharma)]

Hi Alden,
Below is the approach I can think of

1. Receive the file.
2. Loop over the received file
3. Create a batch of 50 or 100 (depending in size of data)
4. Submit the request to SNS.

So in this case if we are going with a batch of 50 and we have total 200 rows. Then we will be connecting to SNS only 4 times.

As of now we don’t have any documented this kind of use case. We will definitely work on this type of use case.

You can also refer the below link.

Handling JSON strings and large files in Integration Server

Description Previous versions of Integration Server provided support for handling JSON content using built-in services for converting JSON strings and files to IData format and back. With Integration Server, version 10.7, this capability is extended using new built-in services for iterating over and extracting data or data fragments from large JSON files. In this tutorial, we will look at how to: Create a simple flow service that converts JSON content to an IData document, transforms the…

This links talk about similar kind of use case implementation on IS on prem.

If we talk about the size of the payload that can be handled by the webMethods .io integration, then it depends upon the platform on which your tenant is hosted.

Refer the below link for same
https://docs.webmethods.io/integration/workflow_building_blocks/payload_and_log_data_support/#gsc.tab=0

Regards
Vikash Sharma