webSphere MQ Adapter: Listener freezes

Hi,

I’m giving support to different clients that have our application installed on a webMethods 6.5 Integration Server environment. One of our clients is seriously experiencing problems with MQ which blocks them to move to production.

The problem:
An MQ Connection, MQ Listener and MQ Listener Notification are set up to retrieve message from the client’s backend to the webMethods IS via a queue.
At a certain moment in time (mostly, but NOT always) at a high load of messages the MQ Listener STOPS listening the queue and ALL messages are blocked on the client’s queue.
STRANGE: the MQ Listener is still at status ‘Enabled’, while NO messages are fetch from the queue anymore.
When I try to disable the MQ Listener the status freezes on ‘pendingDisable’, and I need to force the listener to shutdown. The listener will start up correctly when trying to enable it again.
LOGS: in the server.log a JAVA OutOfMemory error was found, but this didn’t kill the webMethods IS since other non-MQ Listener related processing continued some minutes after this error.

The client is not pleased on this since a manual intervention is necessary to get the MQ Listener working properly again.

Failed Attempts:

  • Installation of all possible fixes for the webMethods MQAdapter 6.0
  • Increased logging and enabling MQ Trace logging → no results
  • Escalated to webMethods Advantage → no result so far
  • Upgrade of webMethods 6.0.1 SP4 to webMethods 6.5 SP2

Client Configuration

  • Solaris 8 (Patch 117350-33)
  • webSphere IBM MQ Server 5.3 (MQ setup: clustered)
  • Oracle 9.2.0.5 DB Server
  • webMethods Integration Server 6.5 SP2
  • webMethods Trading Networks 6.5
  • webMethods webSphere MQ Adapter 6.5

I hope this suffice, but don’t mind to ask for any more information! Thanks a lot in advance!!

Cheerz!

Is the adapter notification setup to run as a “true listener”? Also, what is the “read timeout” setting?

If possible, disable all other IS logging and select only the logging “facility” associated with Adapter Runtime and crank up verbosity to 8 or 9. This may show something useful.

Have you tried having the listener invoke a message notification service directly instead of publishing a document to Broker? Not sure why this would help other than you would be eliminating a publish, subscribe and triggered service invoke.

Mark

Hi Mark thanks a lot for you reply!

I’m not 100% sure of what you mean with “true listener”, but fact is that we have a MQ Connection which is used by the MQ Listener and this Listener triggers a MQ Listener Notification. These 3 components are set up as follows:

  • MQ Connection: WebSphere Transactional Connection. (The client is using a MQ Cluster)
  • MQ Listener: WebSphere MQ Single-Queue Listener Service
  • MQ Listener Notification: Websphere MQ Synchronous Listener Notification

We are not using the Wm Broker. But what I can confirm is that when the situation takes place (= Listener freezes), we are able to execute a Get service on the same MQ Connection as configured for the “failing” MQ Listener and this succeeds. So we eliminated the fact that the MQ Connection itself is causing the problem.

Thanks a lot for the logging tip, I’m going to ask our client to configure it like this.

Bart