We had a production event last week where we lost guaranteed documents during migration of package update to Production. We run IS 6.1 and below is high level architecture
Plenty of guranteed documents published to Broker and more than 2 clients for same documents which retry until successful execution.
After migration we see 20 odd guaranteed document say ‘x’ being published but never came to the clients.
All clients are concurrent, with capacity 10 and execution thread 1.
Below is the procedure used by our admin group to migrate packages
On IS “Maximum Document Retrieval Threads” is reduced to 1 and wait till current number of document retreival threads is to 0.
Deploy the new package using Admin UI.
During the above process we had no transient errors still afterwards we find 30 odd guaranteed documents logged as published but never came to the clients.
It would great if any of you guys can share if they experienced similar data/document loss during package migration which contains triggers for guaranteed documents.
No change to the trigger in the package. We suspend the document retreival on the server before migrating the package. The suspend mechanism used is to reduce the ‘max document store capacity throttle’ to 0%(suspend) and then wait till document retreival threads drop to 0 and then push the package.
Hmmm. It sure seems like the subscription was dropped momentarily, causing published docs to be missed. I’m not sure what happens to triggers/subscriptions under the covers when a package is unloaded and reloaded. Anyone else have an idea?
Below IS behaviour during package reloading is what was confirmed for us from wM support which led to loss of guaranteed documents
“- During the time when you load new release of a package that has the trigger over the existing one, the subscription does get remove, and one the package is successful load, it does add the subscription back. So, if there is document coming during that small gap. you will loose document.” wM Support
This is very bad. There must be way to preserve subscriptions when a package is unloaded. When using IS clustering, deleting a trigger doesn’t delete the subscription. It has to be done manually using Broker Admin. I wonder if there is a way to get this behavior (indicate to IS don’t delete the subscription) without using IS clustering?
Support didn’t give any indication of how this might be accomplished?
wM Support seems to have agreed that this is a bug in case there are more than one IS sharing same client prefix id and not cluster aware talking to same broker. But not a bug for one IS and one broker setting.