How to transform DB data of 1 million records to json and post it to URL

, , , ,
1

Hi Team

We have a requirement where we need to get data from DB and post this to a rest end point.

I will convert the DB data to JSON using the pub.json:documentToJSONString but when I post that huge data I am getting memory exception error. So do we have have any technique to handle this.

One thing i can think off is to use repeat step and put a threshold say 5k or 1k record to process and invoke the http and do this till the total records are completed.

Loop Over Data
Branch
Sequence: $iteration==%threshold%(Here threshold is 1k)
pub.json:documentToJSONString
adjust the json like replacing the end string } with ](as my requirement is sent it as an array)
Repeat
pub.client:http

Thanks
Abhijith

2

Hi Abhijit, yes this could be an option now. Currently the public service pub.json:documentToJSONString puts everything in memory and hence you see the OOM exception when handling large payload.