Emergency eviction of Universal Messaging cluster members

  • Product: Universal Messaging 
  • Versions: 9.0 and higher 

Configuring Universal Messaging clusters is only possible when all cluster nodes are up and running. If the data directory for one cluster node is lost unrecoverably through a disk crash or accidental deletion, you will not be able to reconfigure the remaining cluster nodes.This document describes an emergency procedure that can be used to evict a node from a Universal Messaging cluster without the entire cluster being available.

WARNING: READ THESE INSTRUCTIONS CAREFULLY!! THE PROCEDURE IS NOT REVERSIBLE!

Preparation #

This procedure can only be applied when all nodes in the cluster are STOPPED. Makes ure that all realms are stopped before using this procedure.

You need to evict the lost cluster node from ALL remaining cluster nodes. In this example, let’s assume you have a cluster called cluster1 consisting of three realms umclus1, umclus2 and umclus3. Realm umclus3 has been unrecoverably lost, so you need to remove umclus3 from the cluster configuration on both umclus1 and umclus2.

It is also good practice to make a backup copy of

/UniversalMessaging/serverdata/RealmSpecific/cluster.nst

and realms.nst on every realm before performing this procedure. If you make any mistakes, you can restore this file from your backup.

Evicting the cluster member #

IMPORTANT: Make sure you repeat this procedure on every remaining realm in the cluster and ensure that they all have the same list of known members beforere starting any of the realms.

Cluster settings #

Open a Realm Server Command Prompt on umclus1.

First, check the cluster membership using this command:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nClusterTool -print

This will print out the cluster membership according to umclus1:

----------------------------------------------------
Cluster Name: cluster1
-------------KNOWN MEMBERS--------------------
-------------------------------------------------
Member : 0 , umclus1
----------------------------------------------------
Member : 1 , umclus2
----------------------------------------------------
Member : 2 , umclus3

Remove umclus3 using the following command:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nClusterTool -id=2 -remove

where you use the cluster member number from the -print command in the -id(highlighted above).

Realm removed from cluster

Check using the -print command that the cluster now contains the members you expect:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nClusterTool -print

----------------------------------------------------
Cluster Name: cluster1
-------------KNOWN MEMBERS----------------
--------------------------------
Member : 0 , umclus1
----------------------------------------------------
Member : 1 , umclus2

Remote realm settings #

Next, use the following command to list remote realm connections:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nRealmTool -print
----------------------------------------------------
Local Realm Name: umclus1
Parent Realm Name:
-------------KNOWN REALMS---------------------
------------------------------------------------
Realm (ID, Name): 0 , umclus1
Connection Details (ID, URL, Connect, Multiplex, InterRealm): 0:0 ,
nhp://server1.company.com:9000/ , false , true
----------------------------------------------------
Realm (ID, Name): 1 , umclus2
Connection Details (ID, URL, Connect, Multiplex, InterRealm): 1:0 ,
nhp://server2.company.com:9000/ , false , true
----------------------------------------------------
Realm (ID, Name): 2 , umclus3
Connection Details (ID, URL, Connect, Multiplex, InterRealm): 1:0 ,
nhp://server3.company.com:9000/ , false , true

Remove umclus3 using the following command:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nRealmTool -id=2 -remove

where you use the cluster member number from the -print command in the -id (highlighted above).

Realm removed

Check using the -print command that the cluster now contains the members you expect:

java -DDATADIR=../data com.pcbsys.nirvana.server.apps.nRealmTool -print
----------------------------------------------------
Local Realm Name: umclus1
Parent Realm Name:
-------------KNOWN REALMS-----------------
----------------------------------------------------
Realm (ID, Name): 0 , umclus1
Connection Details (ID, URL, Connect, Multiplex, InterRealm): 0:0 ,
nhp://server1.company.com:9000/ , false , true//
----------------------------------------------------
Realm (ID, Name): 1 , umclus2
Connection Details (ID, URL, Connect, Multiplex, InterRealm): 1:0 ,
nhp://server2.company.com:9000/ , false , true

Repeat #

Now open a Realm Server command prompt on the next remaining cluster member (in this case umclus2) and repeat the above procedure.

The information in this document is provided as-is and with no warranty whatsoever. Software AG cannot be held responsible for the results of any inaccuracies.