We are seeing the below error in IS server.log. We have 12 IS in a cluster. This error happens only on one IS. Whenever this error happens, publishing takes more time ( more than 30 seconds ). I have verified the Broker, Trigger and IS settings and they are all same across all the ISes. We have also seen that publishing takes more time even when the below error doesnt happen. Both the scenarios happens daily for sometime ( any random time ) only on one IS. I have gone through some of the earlier postings over here but couldnt find anything conclusive.
Any help is appreciated.
Thanks
2012-05-07 09:55:53 MDT [ISP.0085.9998E] Exception → java.net.SocketTimeoutException: Read timed out
2012-05-07 09:55:59 MDT [ISS.0098.0036E] DefaultProducer encountered Transport Exception: com.wm.app.b2b.server.dispatcher.exceptions.EndpointUnavailableException: [ISS.0098.9014]
BrokerException: Timeout (112-1450): The request timed out.
2012-05-07 09:56:00 MDT [ISS.0098.0036E] DefaultProducer encountered Transport Exception: com.wm.app.b2b.server.dispatcher.exceptions.EndpointUnavailableException: [ISS.0098.9014]
BrokerException: Timeout (112-1450): The request timed out.
2012-05-07 09:56:00 MDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
2012-05-07 09:56:00 MDT [ISS.0098.0064I] Publishing delayed while outbound store is draining. Service: wm.server.publish:publish
Cause : I believe if IS-Broker connection gets broken,docs will be piled up on outbound store which acts as temp storage area until IS-broker re-connects.Once IS-broker connects back,depending on backlog of documents , publishing will be delayed.
I assume all the 12 IS in cluster are connected to same broker with same shared client prefix , if its the case I would check the dispatch.cnf file size across all the IS.For Instance,If IS1 filesize is 6KB and IS2 filesize is 5KB,copy dispatch from IS1 to IS2 and try bouncing the IS2 for changes to take place.
Hi Kiran,
All 12 IS are connected to their own broker. We have tried cleaning up the outbound doc store, trigger store files and then restarting Broker, IS but still the issue persists.
This error happens sometimes but it always happens on a specific IS / Broker. If we take this IS out of the cluster, the error doesnt happen on any other IS / Broker.
A long shot but you might check hardware issues. Perhaps the NIC is bad in some way. In a previous project it was determined that a port on a network switch was going bad which caused a variety of problems.
I would try taking a backup of current dispatch.cnf , delete it and restart the servers. Re-Configure the IS-Broker connection and restart the servers/sync all docs to broker.