Broker and IS connectivity issue

, , ,
1

Hi All,

We are facing one strange issue in our production server. We could see some connection errors in the integration server logs. checked with network team if there is any blip, but no issues were found from network team. Can some one let me know the possible cause of disconnection between broker and IS?

Suspected some SAN issue as the SAN component that exists with document store sometimes gets exhausted and cause the broker to stall and close all connections. Storage team also said that there were no significant errors during issue occurrence. Please suggest me If I need to check some parameters at SAN level to dig this issue further?
version : 8.0.1

Logs:

2015-09-23 10:40:07 EDT [ISS.0098.0036E] DefaultProducer encountered Transport Exception: com.wm.app.b2b.server.dispatcher.exceptions.EndpointUnavailableException: [ISS.0098.9014] BrokerException: Connection Closed (103-1070): The connection is closed. (mobprodwwd@mwwd-vbrokerprd:6839)

2015-09-23 10:40:08 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:08 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:09 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:12 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:13 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:16 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:17 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:18 EDT [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2015-09-23 10:40:29 EDT [ISS.0025.0025I] Broker Synchronizer initialized
2015-09-23 10:40:29 EDT [ISS.0099.0006E] Unable to disconnect Broker Transport. Exception: com.wm.app.b2b.server.dispatcher.exceptions.TransportException: The connection is closed. (mobprodwwd@mwwd-vbrokerprd:6839)
2015-09-23 10:40:32 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:37 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:42 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:47 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:52 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:40:52 EDT [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2015-09-23 10:40:57 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2015-09-23 10:41:02 EDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish

Can some throw some light on this and help me to find the cause of the issue?

2

Hi Surendra,

can you provide some more details please:

Exact fix level for the broker (awbroker version)
Storage File size for Broker/data/awbroker80/default

From MWS:
Under Messaging → Servers the Utilization Info for the Broker

Is the broker process still running when this happens or is it going down?

Is there still some space available on the SAN or not?
In the first case you can add an additional data storage file (see Broker Admin Guide for details).
In the second case first get the SAN enlarged and the follow the first case.

Additionally you should check if there are documents stuck in the Broker Queues and why.
If the documents being stuck are no longer needed you can delete them from the Queues.
This can be found under Messaging → Clients in the MWS.

Regards,
Holger