Flat File parsing Issue

, ,
1

Hi,

I have developed a flow service which does the file polling and parse the flat file. Once parsing is done it insert the records in DB.

Now there are few bad records in my flat file which i want to notify back to client. How can i identify this records and can collect and notify all these bad record at the end of service?

Thanks

2

"Now there are few bad records in my flat file which i want to notify back to client. How can i identify this records and can collect and notify all these bad record at the end of service? "

what do you mean by bad records and how do you know that they are bad . By bad data do you mean any redundent data or data(that doesn’t comply with your ffschema).can you please provide some more details and sample data

thanks
Anil

3

Hi Anil,

I mean the records which are not as per standard for example if i am expecting date in perticular feild but i receive String or lenght of the feild is too long then in such scenario i want to store all these records and want to send an email back to client with the record details.

Thanks

4

Atul,

Not sure how you figure out whether a record is good or bad - you can use your flat file schema to validate the records upfront and then loop over the errors (returned by pub.schema:validate) to get all the offending records.

If you have a custom validation service instead, initialize a pipeline var (say) errString to blank. Loop over the records, invoke the validation service and if the record is a ‘bad record’, you can create a message appending to the errString with a new line at the end. Do this in each iteration of the loop and at the end of the loop, you can send out an email (pub.client:smtp) with errString as the body of the message.

A better approach to collecting all the errors would be - in each iteration of the loop, after validation service, create a message and append it to errString, if the record is bad. Instead of keeping this errString in the pipeline, you can save it in a repo store using services in pub.storage folder of WmPublic folder. After the loop, you can retrieve the errString from the store and then create an email using errString as the body.

As you can see, there are more than one ways of doing this… choose the one that fits your needs.

~Roh

5

… and yes… if you are using the pub.storage services… in each iteration of the loop, if the record is ‘bad’, you will first fetch (pub.storage:get) the errString already stored in the repo, append the current validation failure message to errString and put the errString back into the repo. I would suggest saving another var (say) errFlag (set errFlag to true) into the repo to indicate that there was at least one ‘bad’ record.

At the end of the loop, first fetch the ‘errFlag’ and check its value, if it is true - it means that there was at least one ‘bad’ record - in this case you will fetch the errString, create and send email. If the errFlag != true, means there was no ‘bad’ record and you don’t have to do the email routine. In either case - do not forget to delete the store before you exit the service. Best to have a finalize block in your service where you delete the store.

Hope this helps,

Roh