Combining realtime amp batch feedback on solution

, , ,
1

Hi. We are trying to solve the following problem in our Enterprise Environment:

Adapter A (JDBC) publishes a document D1 via an “Insert notification” on the database.

Adapter B (File IO) subscribes to document D1 and appends data to a flat file. This process repeats throughout the day.

Once a day this flat file needs to be loaded into our CRM application…but we are worried that as the file is about to be loaded more data will be appended to the file. In theory this process will occur overnight when the users aren’t entering data into the system…but you never know what keeners might come in at 3am to work

Any comments on a solution ? This is 1/2 real-time and 1/2 batch. Our main goal is to avoid losing data when the flat file is being loaded.

Regards,

Wayne

2

Separate the appending from the loading.

  1. A file named something like “D1_temp” collects the docs as you describe.

  2. At the prescribed time, D1_temp is renamed to D1_load and a new D1_temp is created (if needed).

Using a rename is quick and atomic. Don’t try to copy the file from D1_temp to another file as you’ll likely run into concurrency issues.

3

Rob, thanks very much for the feedback.

Regards,

Wayne