UM - Topic consuming too much disk space

, ,
1

Hello,

We’re facing a disk space issue on our Enterprise UM server host.
A single topic - LoyaltyTransactionTopic is consuming 101GB of disk space and its growing daily.
There is an auto maintenance set for the Topic and we’ve tried to purge All events in the UM Enterprise Manager but no luck.

Below is the stats:
[user@serverhost ]$ du -Sh | sort -rh | head -n 15
101G ./data/namedsubqueues/JMS/Kcms/LoyaltyTransactionTopic
23G ./data/namedsubqueues/JMS/Mdm/CustomerTopic
19G ./data/namedsubqueues/JMS/Common/AlertTopic

Attached is the All export config of the UM.
Could you anyone assist us in addressing this issue.

Thanks,

Topic disk space.JPG
Topic disk space.JPG793×662 182 KB

prdent_Allexport.xml (280 KB)

2

Hi,
which version of UM are you using?
And do your subscribers use JMS message selectors?

3

Hello Jonathan,

The UM version is 9.8
For the topic I mentioned above - LoyaltyTransactionTopic

There are 4 subscribers which are durable and have JMS message selectors.

The strange thing is, under namedsubqueues, there are almost 20 .mem files in use

[user@serverhost LoyaltyTransactionTopic]$ pwd
/opt/apps/softwareag/UniversalMessaging/server/prdent/data/namedsubqueues/JMS/Kcms/LoyaltyTransactionTopic
[user@serverhost LoyaltyTransactionTopic]$ ls -lrt
total 112145116
-rw-rw-r-- 1 wmadm wmadm 9550591 Sep 1 14:35 NamedSubscriber639ad2d99212d07f.mem
-rw-rw-r-- 1 wmadm wmadm 783912582 Sep 1 14:35 NamedSubscriber7673df80d53c54c.mem
-rw-rw-r-- 1 wmadm wmadm 1677549456 Sep 1 14:35 NamedSubscriber59d41511adabac7a.mem
-rw-rw-r-- 1 wmadm wmadm 796094481 Sep 1 14:35 NamedSubscriber1654ac9eb79f23c1.mem
-rw-rw-r-- 1 wmadm wmadm 9334953318 Sep 1 14:36 NamedSubscriber77bec62cd37e8676.mem
-rw-rw-r-- 1 wmadm wmadm 9771199130 Sep 1 14:36 NamedSubscriber65b8280c421f5661.mem
-rw-rw-r-- 1 wmadm wmadm 9771199130 Sep 1 14:36 NamedSubscriber62232903703f9345.mem
-rw-rw-r-- 1 wmadm wmadm 9771199130 Sep 1 14:36 NamedSubscriber431d72fdfd86703b.mem
-rw-rw-r-- 1 wmadm wmadm 1041683058 Sep 1 14:36 NamedSubscriber6710aebe8634d654.mem
-rw-rw-r-- 1 wmadm wmadm 9800413075 Sep 1 14:37 NamedSubscriber7516409df5550643.mem
-rw-rw-r-- 1 wmadm wmadm 8166708590 Sep 1 14:37 NamedSubscriber586deecadf86ef40.mem
-rw-rw-r-- 1 wmadm wmadm 8166708590 Sep 1 14:37 NamedSubscriber5739660982ea2632.mem
-rw-rw-r-- 1 wmadm wmadm 8166708590 Sep 1 14:37 NamedSubscriber42a4892713ab3301.mem
-rw-rw-r-- 1 wmadm wmadm 1520932099 Sep 1 14:37 NamedSubscriber613254697a39504a.mem
-rw-rw-r-- 1 wmadm wmadm 8591104599 Sep 1 14:37 NamedSubscriber78ea4ef04a124f04.mem
-rw-rw-r-- 1 wmadm wmadm 8795041495 Sep 1 14:37 NamedSubscriber728d4192bb75c367.mem
-rw-rw-r-- 1 wmadm wmadm 8444567248 Sep 1 14:37 NamedSubscriber641d3d85c32155b.mem
-rw-rw-r-- 1 wmadm wmadm 8444567248 Sep 1 14:37 NamedSubscriber448673f2c3a6b51f.mem
-rw-rw-r-- 1 wmadm wmadm 1782272737 Sep 1 14:37 NamedSubscriber74afbb470b784579.mem

Thanks,
Vinayaka

4

Please check the Realm property (in EM under Realms > Config):
Global Values > SharedDurableFilterBound.
This needs to be set to TRUE. If it is FALSE, then I would expect build-up of data in .mem files.
To activate this, you will need to do the following (which will result in the loss of any in-flight messages on these channels):
• Disable the JMS aliases in all ISes
• In EM, go to the relevant channel for the topic which has filtering, click the Named Objects tab, then click Get Names. Select and Delete the named objects that appear (one for each trigger)
• Re-enable the wM Messaging aliases. This will recreate the shared durables with the right settings.

You do indeed seem to have a lot of namedsubqueue files, which could point to some inconsistency in the internal queues. If the above actions don’t solve the disk space issue, then please open a Support Incident, as they can help you clean up the internal inconsistency.

Hope this helps.

5

Hi Jonathan,

Thanks, I could see the mentioned setting is already set to true.
Global Values > SharedDurableFilterBound=true.

I have opened a Support incident for this, awaiting resolution.

Additionally in EM under Realm Config > Event Storage
MaintenenceFileSizeThreshold is set to 2073741824
I’m assuming this setting will create a new .mem file once the Threshold value is reached. But in our case each .mem file in namedsubqueues are around 9GB.

Thanks,
Vinayaka

6

Maintenance recovers space from files for messages that have already been subscribed. In your case, I suspect the files are orphaned shared durables containing messages that have not been subscribed (and marked for deletion) and hence will not cause files to shrink when maintenance is performed.

7

Hi Jonathan,

As you mentioned about orphaned messages above. Is there any way to delete these messages which has not been subscribed(marked for deletion) from the .mem file.

Regards
Gaurav

8

Hi Gaurav,
Is your question related to the original post or is this a new situation? If the latter, what version UM are you using?
Thanks, Jane.

9

Hi Jane,

Issue we are having is almost same as in the original post where size of .mem file of a particular Topic(durable subscriber) is continuously increasing. Only difference to the original post above and our issue is that there is single .mem file under “namedsubqueues” folder.

UM version is 9.10. Auto Maintenance, SharedDurableFilterBound parameters are also enabled.

Regards
Gaurav

10

Hi Gaurav,

Can you raise an issue? That will enable Support and if necessary R&D to help you.

Thanks, Jane.

11

Hi Jane,

Thanks. I will open a support incident soon for the issue.

Regards
Gaurav

12

Hi Gaurav ,

Have u got any updates from SAG.

You can try perform maintenance from EM.

Regards,
Krishna

13

Hello Jonathan,

I cannot see the property in the config tab of the EM console.

We are using version 9.12 . Could you please let me know if the 9.12 version has different property as such

14

Arun,
the SharedDurableFilterBound is no longer present in recent UM versions, including 9.12 and it is configured to the correct behavior. So if you are seeing unexpected build-up of files on disk, then there is some other reason.
First check whether you have inactive subscribers, such as an IS trigger where the IS is no longer running, or where you have changed the client prefix of the IS. You can check the durable subscribers for a channel in Enterprise Manager and remove any that are obsolete.
If that is not the cause of build-up, then I suggest you open a support ticket.