IS TerraCotta Clustering error

,
1

Hi any thoughts on this error/warning?

System details:

Product webMethods Integration Server
Version 9.0.1.0
Updates IS_9.0_SP1_Core_Fix4
Build Number 1230
SSL Strong (128-bit)

2

Hi Luis,

Is this issue resolved or are you still facing this issue? If its resolved please share the solution.

I’ve similar issue related to Mediator cache – RuntimePolicyCache & MetricAccumulatorCache, it was filling the logs so fast that in a day server.log file was of about 14 GB !! We realized the issue when we scheduled the weekly restart of nodes which are in cluster on alternate days.

As temporary solution, I’ve to recycle Terracotta instances + all the IS nodes of a given cluster and suspend the weekly restart until I get the permanent fix to the issue.

Installation detail at our end

IS - IS_9.5_SP1_Core_Fix3

Terracotta Enterprise 3.7.6

Thanks

3

What is the exact error you are getting same as above error codes?

Also did you open a ticket with SAG support in the mean time and try to resolve it?

HTH,
RMG

4

Hi RMG,

Yes the error is same as posted by Luis.

Issue is raised with global support

Thanks

5

It did got sorted there were some corrupted files in one of the nodes within the folder C:\SoftwareAG\IntegrationServer\config\Caching

Someone on my team accidently modified them.

Maybe it is worth to have a look a those and check with server that has an operational cluster.

Good luck!
Luis

6

Thanks Luis for the come back and update!

7

Hi Luis,

Thanks a lot for the help!

So is the case in our environment, one of the node has diff Mediator cache configuration. I’ve not done the change as this would need a restart and needs to be planned.

Thanks again!

8

Hi All,

Good day . I am facing the exact issue and I checked the files in the caching folder . Files in both the cluster servers look same .

Below is the error message :

[MED.0203.0100W] Thread interrupted when trying to obtain WRITE lock for cache RuntimePolicyCache

The server goes to hang State utilizing 100% memory

Please let us know on what is the solution for it

Thanks in advance

9

Check the below items:

1> Have the latest terracotta fix
2> What is the heap memory allocation on each nodes?
3> In SoftwareAG-IS-Core.xml located under //config/Caching directory check the timeoutMillis for name=“SessionStore” this may be because of network interruption / slowness between Integration Server instance and Terracotta Server Array

10

Yes please check the above settings and see if you can tune a bit based on your environment space allocations!

HTH,
RMG

11

Hi Mahesh,

I am not sure about terracotta fix part.

But the heap memory is 2GB.
RAM is 8GB
DISK SPACE : 40gb
OS: Windows

Timeout mills is set to 45000.

Please advise on the further investigation .

12

Issue still exists? Please have your terracotta fix levels checked.

13

Please open a ticket with SAG as it sounds no more info available from the logs and they can advise you peek in your environment and get all kinds of GC/wrapper logs etc…

HTH,
RMG

14

Hi All,

We did not see the errors after we did the below steps:

Increased heap size and per gen space
Increased the threads count
Made Immediate timeout to false for runtimeCache

Thanks a lot for all your suggestions. It was very helpful

Regards,
Sathyaraj

15

Can you tell which exacly settings did you change and what values?

16

Hi ,

We made the below changes .

permgen space as 50 % of min heap size.
Min Heap size as 50 % of Max heap size
Changed the immediate timeout to false for runtimePolicyCache in the Mediator file present in IS.
Increased thread size to 400 in IS.

Thanks,
Sathyaraj

17

Satya – with the above changes did you fix the issue ?

Thanks,

18

Yes . We did not face any such issues after changing the configuration as mentioned.
In fact, Its been 20 days since we restarted our server after this.

19

Great, thanks for your update.

Thanks,

20

Good deal and sounds like it’s a promising resolution :slight_smile: