Broker starting problem

, , ,
1

Hello,

We are facing problems in one of our production environments.

The current setup is that we have IS running on one instance and have a failover/staging server. the same setup for database.

We had failover on db on Jan3rd. This abruptly shutdown the wM server.

IS Log:

2007-01-03 23:38:42 GMT [ISS.0014.0005C] Shutting down server. JVM Shutdown Hook.
[COLOR=#0000ff][/color]

[FONT=Arial][SIZE=2][/size][/font]After the db restore, we tried to bring the IS back, but we have problems with Broker server running.
1)We have tried to bring the broker manually by running the following command
server_config start -h server:port
But this also failed.
2) we tried to restore the broker data before failover and try to restart the IS, this also failed.
3) we tried to delete the document store and tried to restart broker server.
None of the options worked out.

Here is the error log we are getting everytime when we restart.

2007-01-05 12:56:04 GMT [ISS.0099.0003E] Unable to create Broker Transport 10.196.34.29_6051_AdminClient. Exception com.wm.app.b2b.server.dispatc
her.exceptions.EndpointUnavailableException: [ISS.0098.9014] BrokerException: Broker Not Running (101-1020): The host ‘10.196.50.29’ was found, b
ut no Broker Server is running on port 6059 of that host.

2007-01-05 12:56:04 GMT [ISS.0099.0003E] Unable to create Broker Transport wmedsp05_wm1__DefaultClient. Exception com.wm.app.b2b.server.dispatche
r.exceptions.EndpointUnavailableException: [ISS.0098.9014] BrokerException: Broker Not Running (101-1020): The host ‘10.196.50.29’ was found, but
no Broker Server is running on port 6059 of that host.

2007-01-05 12:56:04 GMT [ISS.0099.0003E] Unable to create Broker Transport 10.196.34.29_6051_RequestReplyClient. Exception com.wm.app.b2b.server.
dispatcher.exceptions.EndpointUnavailableException: [ISS.0098.9014] BrokerException: Broker Not Running (101-1020): The host ‘10.196.50.29’ was f
ound, but no Broker Server is running on port 6059 of that host.

2007-01-05 12:56:04 GMT [ISS.0099.0003E] Unable to create Broker Transport 10.196.34.29_6051_SyncClient. Exception com.wm.app.b2b.server.dispatch
er.exceptions.EndpointUnavailableException: [ISS.0098.9014] BrokerException: Broker Not Running (101-1020): The host ‘10.196.50.29’ was found, bu
t no Broker Server is running on port 6059 of that host.

Problem is when it start up, it is looking into failover server 10.196.34.29 which is wrong. Iam not sure from where wM IS picks up the configuration.

Appreciate any help

Regards
[COLOR=#0000ff]Sai Arun[/color]

[FONT=Arial][SIZE=2][/size][/font]

2

You mentioned that IS was configured in some way for failover, but did not provide details.

You did not indicate whether Broker was installed on one or both servers or if it was configured in high availability mode. You did not indicate whether your crash affected the server(s) on which Broker was running (it certainly appears to be the case).

Details?

3

The dispatch.cnf file holds the configuration for the connection to the Broker. If you have IS set up in an active/passive OS cluster, the dispatch.cnf should be the same on both nodes.

From your error messages, 10.196.50.29 is your Broker Server host. 10.196.34.29 is the IP address of one of your IS hosts. When IS connects to a Broker, it connects to named queues (clients). For many of the queues, the IP address and primary port of IS is used as the name in an attempt to avoid name collisions (e.g. 10.196.34.29_6051_RequestReplyClient).

When setting up IS in a failover cluster, it is typical (actually, required) for both instances to have the same dispatch.cnf, and thus the same IP addresses for the client names. So, the reference to the “failover” IP and port (10.196.34.29_6051) in your message list is probably right.

The root issue is the Broker Server on 10.196.50.29 is not running or is not listening on port 6059.

I take it that there is noone available who set up this configuration and can help you restore service?

Side question: Usually IS is not set up in a hardware/OS cluster in an active/passive mode. This configuration isn’t usually supported by wM. Have you considered using a load-balanced cluster instead?

4

Forgot to mention:

I assume 10.196.50.29 is the virtual IP of the Broker Server cluster. Whichever node “owns” that IP at the present time is the node that needs to have Broker Server running. I would think your cluster configuration and scripts would have taken care of that automatically. Certainly a restart of the machines should also have taken care of that–if not, then it seems that the cluster may not be configured quite right.

5

old post