IS loses connection to broker every minute

, ,
1

Product/components used and version/fix level:

Broker 10.3
Integration Server 10.3
latest fixes applied to both IS and Broker but non from Extended Maintenance support.

Detailed explanation of the problem:

We have 2 integration servers and 1 broker installed on same node. From the worker I get the errors below every minute.

2024-05-09 11:33:11 MDT [ISS.0098.0042I] (tid=1319) Successfully reconnected to Broker. Stopped polling
2024-05-09 11:33:41 MDT [ISS.0098.0041C] (tid=1319) Unable to connect to Broker. Starting to poll
2024-05-09 11:33:51 MDT [ISS.0025.0025I] (tid=1319) Broker Synchronizer initialized
2024-05-09 11:33:51 MDT [ISS.0098.0042I] (tid=1319) Successfully reconnected to Broker. Stopped polling
2024-05-09 11:34:21 MDT [ISS.0098.0041C] (tid=1319) Unable to connect to Broker. Starting to poll
2024-05-09 11:34:31 MDT [ISS.0025.0025I] (tid=1319) Broker Synchronizer initialized
2024-05-09 11:34:31 MDT [ISS.0098.0042I] (tid=1319) Successfully reconnected to Broker. Stopped polling
2024-05-09 11:35:01 MDT [ISS.0098.0041C] (tid=1319) Unable to connect to Broker. Starting to poll
2024-05-09 11:35:11 MDT [ISS.0025.0025I] (tid=1319) Broker Synchronizer initialized
2024-05-09 11:35:12 MDT [ISS.0098.0042I] (tid=1319) Successfully reconnected to Broker. Stopped polling
2024-05-09 11:35:42 MDT [ISS.0098.0041C] (tid=1319) Unable to connect to Broker. Starting to poll

from the other one I get a slightly different version of the same error.

2024-05-09 11:38:05 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:05 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:05 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:06 MDT [ISS.0098.0041C] (tid=1374) Unable to connect to Broker. Starting to poll
2024-05-09 11:38:16 MDT [ISS.0025.0025I] (tid=1374) Broker Synchronizer initialized
2024-05-09 11:38:16 MDT [ISS.0098.0042I] (tid=1374) Successfully reconnected to Broker. Stopped polling
2024-05-09 11:38:45 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:45 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:45 MDT [ISS.0098.0038C] (tid=1361) ***_DefaultClient handling invalid Document. Exception: com.wm.app.b2b.server.dispatcher.exceptions.InvalidDocumentException: [ISS.0098.9006] No matching Document Type exists for Adapter::error 
2024-05-09 11:38:46 MDT [ISS.0098.0041C] (tid=1374) Unable to connect to Broker. Starting to poll
2024-05-09 11:38:56 MDT [ISS.0025.0025I] (tid=1374) Broker Synchronizer initialized
2024-05-09 11:38:56 MDT [ISS.0098.0042I] (tid=1374) Successfully reconnected to Broker. Stopped polling

This is happening on only 1 node, and I couldn’t make it stop no matter what I did.

I tried syncing the document pub.publish.notification:error, it doesn’t return an error if I do it when it is connected. But it doesn’t fix the issue. It complains about the connection when the connection is lost.

Checked dispatch.conf files of both ISs and verified they have different client ID.

Tried to let it fix itself. It has been like this for at least a day. The node itself is not receiving any traffic right now.

Restarted both ISs and Brokers and the OS itself.

Also tried to delete the document from MWS broker admin. It is a system defined document. It doesn’t let me delete.

What am I missing here?

Edit: It has spread to another node. We are still investigating the issue.

2

If the Client IDs have the localhost IP, try changing them to the actual hostnames instead.
If that doesn’t work, try backing up the config file and recreating a new one.

Do you know what had changed, for the issue to appear all of a sudden?

KM

3

Nothing changed. The node itself has 2 IS and 1 broker so it is connecting to its own broker. They are using the hostname to connect, not the IP.

4

From the logs, there appears to be some contention since both the nodes are constantly reconnecting. This usually happens with duplicate client IDs, but you’ve taken care of that.

Another possibility, from historical incidents, is the client timeout value (default is 30 seconds). Your IS nodes are on the same machine, so this is unlikely too. To narrow down the issue, you can give this a try setting the timeout value via a Java service which you MUST configure as a startup service - see setDefaultClientTimeout here (link).

Did you try recreating the dispatch.cnf files? Both the IS nodes must be offline, then you backup and remove the file, and start the nodes which will create the file again.

As a last resort, try reinstalling Broker and if possible, have the IS nodes on different machines.
Nothing else comes to my mind - perhaps others have ideas.

KM

5

What happens if only one IS instance is running?

Is there a difference between IS1 and IS2?

Does the startup order of the IS instances make a difference?

6

It doesn’t make any difference when I change the order of the IS, or let only one of them run at any given time. They produce the same error,

This part of the error is only pushed to the same server. It doesn’t move to the other one if I change the order of sturtup. The rest is the same.

7

This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.