IS Clustering to be avoided

, , ,
21

Hello Rob,
I searched the documentation regarding PRT internals, there are not many details. The hint I’ve is that when these process internal documents are processed, the PRT inserts an entry into the WMPROCESS table with the physical address of the server. This happens for each step transition to a different logical server.

I’m just wondering whether can we configure the non-clustered Integration Servers to share the same repository (this has to happen without enabling clustering).

Kind Regards
Sivaraj Lenin

22

Yeah, I knew about publishing and the WMPROCESS table update for the step transitions. What’s unclear is why an IS cluster would be needed in this scenario. It is my understanding, which may be wrong, that the PRT tables can be shared among multiple IS instances without IS clustering. The PRT tables are not part of the repository.

The repository would be used when a trigger has a join condition. I looked through WmPRT package and unfortunately the trigger conditions are not exposed. Based on your observations though, it seems a safe bet that a join is used.

Sharing the repository is pretty much what IS clustering is all about. So if you share the repository, you have an IS cluster.

23

Hi, I am using JMS Adapter with IS clustering. The requirement here is serial processing should be used strictly. The data flow goes like. Source system calls a web service hosted on IS → web service puts source data as JMS message on the queue → JMS notifier notifies to Broker → Trigger invokes Flow Service → Flow service submits data to RDBMS. Two IS are clustered and one JMS queue is configured. For example, web service has 15 msgs as input. Those 15 msgs are put on JMS Queue. JMS notifier publishes 15 msgs to the Broker. As IS clustering is used, we have trigger subscribing to those document on each IS. so 15 msgs will be divided and sent to both the IS in cluster. In this scenario, how to guarantee that messages will be processed serially as they put on the JMS queue ?

  • Nilesh
24

There can be no parallel processing at any point in the integration.

The source system must be single threaded, calling the web service one at a time and waiting for a positive response before submitting the next.

The IS hosted service publishes the document (I assume you’re just using pub.publish:publish) to the Broker. This service should not create threads in any way and probably isn’t unless you’ve done something explicit.

The Broker will deliver documents in “publisher order” if the triggers are configured as serial. This means only one of the IS instances will get the documents as long as there are more in the queue (see the pub/sub guide for a better description). If the IS that is processing fails, the Broker will notice and give that document and subsequent documents to the other instance.

The IS service invoked via the trigger must not create any threads.

If you follow these, your messages will be processed in order. Note that IS Clustering isn’t needed for any of this to happen. The IS instances simply need to use a the same Broker client prefix to get this failover capability.

25

Anyone successfully implemented JMS adapter polling notifications on an IS cluster such that both nodes in the cluster can poll a JMS topic using a durable subscriber?

So far, no matter how I configure it, I get a “[BRM.10.2002] JMS: Client ID “” is already in use.” error when I enable the second polling notification.

I was getting this same error before spending a full day configuring an IS software cluster. Now I’m getting the same error with the cluster. What a waste of a day.

Note that a multi-node Websphere Application Server cluster in which each node in the cluster is running multiple message-driver beans can subscribe to the same JMS topic using a durable subscriber when the topic has been configured with SharedState=“true” and the wmjms.properties file is configured with the com.webMethods.jms.clientIDSharing=true property and placed in the WAS classpath.

If this can be done with Websphere, shouldn’t you be able to do the same with IS JMS adapter polling notifiations?

Mark

26

Page 30 of the JMS Adapter User’s Guide states:

So, this just once again affirms that there is almost no reason whatsoever to consider using an IS cluster.

Still trying to solve this by putting the “clientIDSharing” property in the JMS Adapter classloader’s path. No luck so far.

Mark

27

I wrote a Flow service to load the com.webMethods.jms.clientIDSharing=true property and made it a startup service.

I then configured the broker settings on each IS to use the same client prefix, confirmed that the JMS topic was set to SharedState=true, enabled the adapter connections and JMS polling notifications and published a test message to the topic.

Only one IS node picked up and processed the JMS message. Mission accomplished. No IS cluster needed (or wanted).

Mark

28

What about Chapter 1 - page 10 “What is Clustering / reliability”? It says:
“… Clustering makes it possible to deliver ‘exactly once’ execution …”
How should I read this?

Rob, I am interested to know. Is it correct that you would use a third party load balancer (which has the advantage of single IP/hostname for the cluster) and not use IS clustering at all? Assuming one doesn’t need the session/transaction management, but is interested in the “execution only once” of messages.

29

What about the scheduled services?

30

It is only needed for scheduled services if you want to schedule the same jobs on multiple IS instances and want the scheduling to be coordinated across servers.

Scheduled jobs and JDBC adapter polling notifications are two cases in which the extra complexity of the IS software cluster might be warranted. I would look for ways to design that need out of my solution if I had any choice whatsoever.

Extra complexity + low to no value added = bad idea.

31

The (inverse) context of the thread is “when to use IS clustering” to get the features associated with high availability. Scheduled services are not such a feature. One doesn’t introduce clustering to get the ability to run scheduled services. Rather, they are a facility that can be impacted by the introduction of any sort of clustering.

As such, one needs to consider the impact to scheduled tasks when designing a highly available solution. For example, scheduled services are supported by “IS clustering” but if one uses only a load-balancing cluster (and not IS clustering) then an approach must be devised to ensure that services that should be running only on one server at a time do so. One approach is to use an external job scheduler such as AutoSys.

32

Thanks Rob, I understand your point of view. However, I have the opinion that scheduled jobs should be part of an HA environment. The timing of when particular jobs run is critical for our business. Introducing an external scheduler like AutoSys or even simple Unix crons, not only makes deployment a lot harder, it also introduces another single point of failure.

I haven’t used IS clustering yet, but what is it that makes you guys want to move away from it? Does it introduce lots of complexities/problems? or do I missunderstand you on this?

33

I think most of this preceeding posts in this thread are focused on the reasons not to use IS clustering. In short, IS clusters add lots of complexity and very little value.

Scheduling jobs in IS rather than a robust, enterprise-class job scheduler is counter-intuitive and, IMHO, usually only done because of lack of interaction between developers building integrations and the operations staff supporting them. Sure, its convenient… for developers anyway, but that’s hardly a sufficient argument.

Most of my clients that were using IS clusters believed they were getting far more value than was actually being delivered (e.g. failover and superior load balancing). Many of the challenges they were having with ensuring a high degree of IS stability stemmed from the added complexity of an IS cluster (or reverse invoke servers).

So, as I always say, “simple is good”. This is especially true when additional complexity doesn’t add value.

Mark

34

Let me rephrase that a little: scheduled tasks should operate correctly in an HA environment. As I stated earlier, one doesn’t introduce clustering to get the ability to schedule tasks. Rather, one introduces clustering to provide HA features (failover, load-balancing, etc.). One must consider how scheduled tasks, JDBC notifications, flat file polling ports, DB access, etc. will behave in the cluster.

You can certainly use IS clustering to get scheduled tasks to operate as desired (only on one instance at time) in the cluster but that isn’t the only option. In addition to the external scheduler that I mentioned, another approach is to enable the tasks only on one instance. If that instance fails, manually start the tasks on the other instance.

Yet another approach is to implement a “poor man’s” cluster-aware facility (I’ve done this in a couple of places)–Modify your scheduled services to check a config file. In that file is a list of the IS instances, in order or primary to secondary, etc. When the scheduled service runs on a given instance, it loads that file. If the first name in the list matches its own name, it continues. If the first name doesn’t match, then it pings the other IS instance. If the other instance responds, the service stops. If the other instance does not respond, the service checks the next name in the list. When it finds its own name and all instances that are above it in the list have failed to respond, it continues to run.

I’m sure there are other approaches that could be followed.

This is an argument for, not against, using a robust scheduler such as AutoSys. For example, the scheduler within IS does not track what it does. If the IS environment was down at noon, and a service was supposed to run at noon, that service will be missed that day. AutoSys detects past jobs that have been missed and runs them.

The deployment becomes an AutoSys team task, not an IS team task. “A lot harder?” I think you may overstate the case a bit. For the “single point of failure” one needs to implement clustering for AutoSys too.

Don’t get me wrong, committing to a robust environment for AutoSys is not a trivial undertaking. But neither is committing to IS clustering and the repo server.

The first few posts in this thread point out the issues. And Mark summarized it quite nicely. IS clustering has been successfully used–but it comes with a cost and it doesn’t provide nearly the level of functionality that people usually attribute to it.

35

Folks,
Interesting discussion, going forward I would suggest not wasting your energies with any of these options presented in this thread. By the end of this year wM is throwing out the current clustering using repository and are incorporating the offering from tangosol. If my resources are correct they will be using their coherence product. Here is the link if anybody wants to review the product.
Excerpt from [URL=“In-Memory Data Grid | Oracle”]http://www.tangosol.com/coherence-overview.jsp[/URL]

"Tangosol Coherence enables in-memory data management for clustered J2EE applications and application servers. Coherence makes sharing and managing data in a cluster as simple as on a single server. It accomplishes this by coordinating updates to the data using cluster-wide concurrency control, replicating and distributing data modifications across the cluster using the highest performing clustered protocol available, and delivering notifications of data modifications to any servers that request them. Developers can easily take advantage of Coherence features using the standard Java collections API to access and modify data, and use the standard JavaBean event model to receive data change notifications.

Tangosol Coherence provides coherency of data in a cluster, ensuring data integrity even for cached data. By managing data safely in the application tier, applications can scale much higher without significantly increasing the load on shared enterprise resources, such as database servers. This enables organizations to much more accurately predict the cost of scaling an application up to the enterprise, and provides a higher degree of confidence that the application will continue to perform well as it scales up."

regards,
Adnan

36

The title of the thread “IS Clustering to be avoided.” If wM throws out the current clustering facilities within IS, then those that follow the original advice of this thread are in good shape! :slight_smile: Even so, users of 6.1 and 6.5 will be around for some time, so a discussion of alternatives isn’t a complete waste.

I would submit that the option of using a load-balancing cluster is viable, indeed probably necessary, even when (if) Coherence is integrated within IS.

The description is interesting. I wonder how wM wiill leverage it? After all, IS is not an “application server.” Its focus is different than the applications one typically finds implemented within WS, WLS, JBoss, etc.

If we look at the clustering features currently provided:

  • Session data management (when mulitple interactions with IS for a single transaction is needed)

  • To support join conditions for Broker subscriptions

  • Coordinated scheduled tasks–a task is only ever active on 1 server

  • Coordinated ART notifications–a notifier is only ever active on 1 server

Each of these would presumably be retrofitted. I assume, too, that additional facilities would be modified to support clustered operation.

TN has the notion of a “TN cluster” which doesn’t require IS clustering. TN instances share a common DB for the run-time data, and can directly communicate to tell each other “refresh your meta-data cache.” This would seem to fit in with what Coherent does–assuming wM updates TN for this!

I assume there will still be a persistence facility of some sort. It would seem to be a prerequisite for a robust solution.

37

Rob u miss interpreted “going forward” and “wasting your energies”. Both are mutually exclusive :). If you can live with current IS clustering there is no point reinventing a wheel…

38

I’m confused. Can you elaborate on your clarification?

39

We are implementing here a HA architecture.

For that we defined several IS with Microsoft NLB and without IS cluster.

But we have other integrations that are trigered by:

  • Schedule Tasks
  • Adapter Notification
  • Broker Triggers

how can I make this things continue to work if one IS crashes without IS Cluster ?

I read that for Broker Triggers I can do that defining the same broker prefix in two IS.

But what do you suggest for

  • Schedule Tasks
  • Adapter Notification
    ?

Thanks

João Caseiro

40

I guess I missed this back when it was posted.

It is indeed correct that I’ve implemented solutions using a third party load balancer without using IS clustering.

IS clustering doesn’t “make possible” exactly once execution. It certainly eases some of the issues–like scheduled tasks that should be run on just one server at a time, or adapter notifications that need to be processed just once–but it isn’t a requirement.

Much of the clustering guide overstates the capabilities. Well, actually it really just has descriptions that are easily misinterpreted. For example, the load balancing feature, which has now been deprecated, has as a footnote “oh by the way, load balancing works only for http.”

IS clustering is scheduled to have an update soon, rumor has it. I look forward to seeing what changes will be available.