We are using webmethods.io integration cloud to get files off our SFTP folder, take a file (most commonly a large CSV, ~100k-200k rows), normalize the records to our JSON schema, and then submit the records to Amazon SNS. All of that job is currently handled via a flow service. We currently have a flow that submits each row on the file as an SNS message, but it’s too transactionally expensive opening up the SNS connection for each row. Does anyone have recommendations to make this flow possible?
- Chunking: taking the CSV file and parsing it into smaller batches that we load as a message to SNS. So for example, we take a 1000 line file and break it into 4 250 row batches, and send each batch to SNS as a single message (versus sending 1000 messages, 1 row per message).
- Better flow service performance: are we missing basic functions or means of holding open SNS connections to send large volumes of messages more efficiently, without the cost of opening/closing the Amazon connection for each row?