Unzip and read large file in IO

,
1

Hi
I’m trying to implement below use case in WMIO Workflow

  1. Download zipped file from external API (Zip file contains only one file, size can be like 50-200mb)
  2. Unzip file within workflow
  3. File contains JSON array object. Read file content and then store into SQL DB each object.

So far managed to download and unzip the file using “Unzip” activity. This extracts unzipped file on local file store within IO (like /home/sagadmin/flow/user_data/vbid19d49a4ee7a5d7803a82e6552be2c4ffae6f454a7baa/unzipped-records/). Now I’m not able to read the file as the size is too big for Read File activity - { [Message Size Too Large] }.

What are my options here to solve this issue. Will a custom nodejs code will help or are there any better way to implement the solution in IO?

TIA

2

Hi,
You can use the action “Open readable file stream” to do what you want. The error message that you are getting is probably due to the testing feature in the editor. To build the steps, try using a small file to avoid the browser limits. Once everything is hooked up you should be good to run it against the proper sized file.
regards,
John.

3

Thank you for your response, John.
We found alternative client APIs that allow us to avoid storing zip files.

I wasn’t aware of the UI limitations in displaying the response. It’s good to know about this and I’ll definitely keep it in mind while working on similar workflows.

I’m curious about how to use the ‘Open readable file stream’ feature. If you have any sample workflows, please share.

4

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.