Keeping huge amount of documents waiting for other service in UMS

, ,
#1

Greetings,

For given situation:

  1. Multiple messages are sent to IS
  2. Based on that messages JDBC adapter should be called(thousands inserts per 1 incoming message), but only after other service gives green light for it (could be 15 minutes from receiving first input message). There could also be a case when few input messages are linked(by some parameter in message), and all those inserts should be called at the same time.

Client insists on having all of those thousands messages published via UMS, and waiting there for a service which will give green light to call DB.
I am wondering how to deliver that, should I publish the documents without creating a trigger and then create some java service which will take documents from the queue/topic? Should pub.publish or pub.jms services be used for that?

I also have doubts if that’s a proper way to handle the situation(having tens of thousand messages stuck in queue for minutes, what if the ‘green light service’ fails, etc).

Would appreciate your input on that issue.

#2

I am not sure if UMS is the place to keep the records waiting in this scenario. You could probably use a Staging table with a flag column in it that can get updated to ready only when the other service gives the green light. You can setup an update notification on the staging stable and pick the records and process them when ready.

1 Like
#3

I will second Akshith on that. We had a similar use case for a HR Transformation project - where we had to first move over data from EC system to a central hub and hence all this different data (what they called portlet) - where supposed to be transformed into legacy components, so as to ensure feed to downstream system are generated for fields as it was being generated from the old system. Have we taken the approach to use UM and hence populate the legacy table (in certain case - we would have to setup trigger execution through subscribing multiple documents [as and condition] together) - we would have landed in similar situation of UM getting overload and could have had a degradation of performance too.

So, I believe in such situation breaking them in modules i.e. loading in stage and then processing from stage to actual tables works out much better. To speed of the process of loading though - you may use Concurrent approach and hence time the process and execute the other piece in the same flow or after the whole workflow of loading into stage is over (as the ask be).