Serial processing with MQ Listeners in Cluster

, ,
1

Hi All - Here is what we need to do: As messages are put on MQ, we need a multi-node IS cluster to pick them up off the “GET” queue.

We are using an asynchronous MQ Listener which publishes a document to the broker for every message on the MQ queue we’re listening to. The MQ Listener polls the queue every 60 seconds.

Does anyone know if there’s a way to ensure that each node puts these Notification Docs on the broker in a serial manner? In other words, if wM node1 picks up MQ msgs 1-10 and puts them on the broker, then node2 checks out the queue and notices MQ msgs 11-20 and soon afterwards, wM node3 checks the queue and notices MQ msgs 21-30 … is it possible that MQ msgs 21-30 Notif docs can be published to the broker BEFORE MQ msgs 11-20?

I know this sounds confusing, so if it’s unclear, please let me know and I’ll rephrase the problem.

2

There is an option called Priority that can be set for each messages in MQ. If you set higher priority for the messages while placing in queue, which you want to process first, those will be processed leaving behind the other messages which will be done later…

3

Thanks for your response Senthil. I was able to confirm that the messages are queued up fine in MQ.

My belief is that they’re out of sequence by the time wM picks them up off MQ and places them on IS’s client queue (i.e. the subscribing trigger).

Here’s the scenario:

1 - Suppose MQ puts up to 500 messages on the Listener queue every 10 seconds
2 - IS node1 MQ Listener picks up first batch (let’s say there are 400 this time) and puts the 400 messages on the trigger.
3 - IS node2 MQ listener thread runs 2 seconds later to get the next batch (let’s say there are only 5 messages in 2nd batch).

Couple questions:
1 - Is there some limit of how many messages a given listener thread can pick up?
2 - Is it possible that IS node2’s MQ listener thread puts its 5 messages on the IS trigger before IS node1 does?

Thanks!
Rajesh

4

Hi Rajesh,
IS Node 1 and IS Node 2 are in clusters.

In general, we cannot say, how many messages a listener picks and which listener will pick; If listener 1 picks 100 messages, other listener 2 in cluster will pick the remaining messages and starts processing in separate thread.

To answer your questions.

  1. No. There is no limit on how many messages a given listener will pick.
  2. Yes. As they are running on different threads in different IS.

Regards,
Senthil

5

In SAP adapters, while configuring a listener, there is a field called ‘Number of Threads’ which tells ‘The number of simultaneous incoming RFCs that this listener can handle’.

In MQ, there is no field like this while configuring listener. And thats the reason, there is no specified limit on how many messages a given listener will pick… To my knowledge, there is no concept of Listener thread in MQ to be explicitly mentioned…

In order to maintain the sequence,

  1. Enable the listener only in one node in clustered environment. This ensures, there is no duplicates and, the messages are picked up by webMethods in sequence as MQ operates in FCFS.

  2. Listeners published the Messages to the broker in sequence, and broker client queue maintains the sequence. IS triggers from clustered environment picks the messages and processes it.

Thanks,
Senthil

6

MQ Listener Notification documents out of sync on IS client queue

URL: [URL]http://advantage.webmethods.com/article/?id=1614337859[/URL]

Issue
Running IS v7.1.2 with Broker 7.1.2 on Solaris 10.

In a Production environment all 3 IS’s have a “client prefix” and “client group” set to the same value.

The interface is designed to pick up messages off an MQ queue via an asynchronous webMethods MQ Listener notification.

A serial trigger subscribes to this MQ Listener Notification document. The trigger’s capacity and refill are both set to ‘1’ since the requirement is that the MQ messages must be processed in the order that they are placed on the MQ queue.

The behavior observed is that messages are being processed out of sequence.

For example, if there are 50 messages on the MQ queue, the IS trigger processes message number 20 before processing message number 8.

Is there a way that documents can be processed in the order they were placed on the MQ queue?

Resolution
The MQ Adapter, in a clustered or non-clustered environment, does not guarantee the order of messages received from WebSphere MQ.

With 3 IS instances involved, each one of them is listening to the same MQ, and message ordering cannot be guaranteed at the trigger level. Each of the IS instance will pick and process the message at its own pace and publish the document. That is why message ordering cannot be enforced.

It has to be the application processing the notification documents that must handle any reordering of messages if it’s required.

You can see this discussed on page 60 of the webMethods_WebSphere_MQ_Adapter_User’s_Guided, under the section titled �WebSphere MQ Adapter in a Clustered Environment�, sub-paragraph ‘Ordering of Documents’.

“Neither the WebSphere MQ Adapter nor WebSphere MQ guarantee the order of messages received from WebSphere MQ. The application processing the notification documents must handle any reordering of messages that is required.”

7

Thanks Senthil. That’s the SR I opened with SAG. I must admit that I was surprised that the MQ Adapter, Broker and IS can’t do this :frowning:

8

Rajesh,
Probably not what you want to hear but I think you have some mixed use cases for IS instances with this requirement. Sounds like you have classic batch processing to do, serial FIFO, order matters. Yet your IS instances have been clustered who’s sole purpose in life is to share the love with each other. :biggrin:

Not to beat a dead horse but clustering generally creates more issues than it solves. Specialized IS instances are easier to support and provide a lot of other advantages over webMethods IS clustering technology:eek: which is rarely needed in my opinion and experience (upgrades, separation of concerns, ease of maintenance, troubleshooting etc).

A “batch” IS instance with active/passive clustering can accomplish what you are trying to do while a pair or three of externally load balanced IS instances can handle your inbound client type stuff.

If you still need true active/active clustering meaning your scheduled tasks and your cluster aware adapter services need to share processing power across multiple instances and you need immediate pickup of IS initiated transactions across the other nodes then you can still do that but with your batch services separated out onto their own instance. The batch IS instance with MQ Adapter and serial IS trigger will preserve your FIFO requirements. Just make sure your MQ queue settings have FIFO set, which is the default.

just some thoughts