We need to pull a large file (40MB) from FTP and then process it. I am able to pull it from FTP (takes about 30 mins) using stream option. The file has just one header record and the remaining records (350K lines) are detail records. I couldn’t use ffiterate to convert to values as there is nothing to iterate upon. I wrote a java service to filter the 350k records and bring it down to about 60k records (7MB). Then I still need to pass it through convert to values, loop through 60k records and publish the final doclist to broker because of the existing integration scenario. The whole process is taking about 2.5 hrs. Is there any better way to convert to values such a large file?