Duplicate Documents in UM

, ,
1

Hi Gents,

I am facing a strange issue with UM. Please let me know if anyone of you have faced it.

Scenario:

Scenario 1

  • When i am publishing 1000 Messages to UM and subscribing it serially then 1000 records are being inserted successfully. – OK
  • When i am publishing 1000 Messages to UM and subscribing it concurrently with the below properties following things happens:
    • Setting Duplicate detect to true → No issues and 1000 records are being inserted into DB. --OK
    • Setting Duplicate detect to false → Then 2500-3000 records are being inserted into database. - NOT OK

Have tried with less number of records as well…tried inserting 5 records and faced the same issue mentioned Above.

Scenario 2

  • Publishing 1000 messages with duplicate content to UM and subscribing it serially then 1000 records are being inserted successfully. - OK
  • Publishing 1000 Messages to UM and subscribing it concurrently with the below properties following things happens:
    • Setting Duplicate detect to true → Then 450-700 records are being inserted into DB. - NOT OK
    • Setting Duplicate detect to false → Then 1500-2000 records are being inserted into database. - NOT OK

I have two Questions here…

  1. Can’t we avoid duplicates without using detect duplicate property of trigger in UM? I don’t want to use it because for heavy usage (Millions of transaction/Hour) it will have performance impact.
  2. Is UM capable of detecting the duplicate content of message? I hope not? because this will violate business requirements.

Note: I have applied all the latest fixes and also i am using UM over JMS.

Thanks in Advance,
Regards,
Syed Faraz Ahmed

1 Like
2

Syed,

You say you are using JMS. Are you using topics or queues?
If topics, are you using durable subscribers?
Also, what UM version are you using?

3

Hi Jonathan,

I am using Topics, UM version is 9.9/IS version is 9.9.
I am not using Durable subscriber.

Regards,
Syed Faraz Ahmed

4

Syed,
if you are using topics without durable subscribers, then the behavior you are seeing is expected and would be the same with any JMS provider - not just UM.
If you have a concurrent trigger, it looks to UM as multiple subscribers, so each message can reach multiple threads in the trigger. By using duplicate detection you are just hiding this phenomenon, but it is very inefficient.Also, the use of a topic without durable subscribers does not guarantee that a message reaches a subscriber.

If you want to be sure that things arrive and that they normally arrive just once, also with concurrent triggers, then you need to use a durable subscriber. This can be configured in the IS JMS trigger. Also make sure that the Connection Factory you use has Shared Durable enabled, which is required to allow concurrent access to the DS.

1 Like
5

Thanks a lot for detailed explanation.

I will re do the test cases with Durable subscribers and let you know the results.

Regards,
Syed Faraz Ahmed

6

Everything is working as expected. Thanks a lot for your help.

I have one more query will open a new thread for that.

Regards,
Syed Faraz Ahmed