Flatfile--> Limitation -> Need to write sequential to flat file

, ,
1

We have requirement to send the database records from DB and convert it into flat file and send to other system using webMethods.

But the data has to be pushed into a single flatfile, there are 25,000 records in a batch. This needs to run daily.

We are thinking to push the data using the same batch number, which is inserted into the buffer table (notification is configured on this buffer table). The publisher will read every 500 records and publish the canonical to the broker.

Now the subscribe read this canonical and converts into the flat file using ffschema and append to the batch file.

Till this point we are fine, wat is going to happen if the publisher published four canonical to the broker, then we will have 4 subscribers writing/appending to the same flat file.

We need a solution,where the subscriber should need to generate a tag/ some indicator that the file writing is complete.
2. Or making the subscriber to read the canonicals sequential, do we have any mechanism in webMethods?
3. Can subscriber go back to the DB for any cross reference?

Please help us on this?

Thanks
Khan

2

U can make trigger processing mode as serial instead of concurrent.this may solve ur prob.

3

The easiest approach would be to not use the Broker at all. Unless doing so adds value to the process, I’d exclude the use of Broker from the solution. The use of pub/sub here seems unnecessary.

Since this appears to be batch oriented, you can probably also eliminate the use of a DB notification. Just schedule a service which reads from the DB and writes the file.

4

since you said it is going to run daily (i assume once or few times a day), best way is to schedule the job, this will save buffer tables/triggers/notifications etc.
As Rob suggested, there is no point in using pub-sub, create the flat file with in the same servcie. May be you can set maximum records to be fetched per DB query and write to flat file; continue to query database for all records. - this will save publish/subscribe operation.