Anyone having problems with Broker 652?

, , ,
1

Hey all - we’ve been really struggling with an upgrade.

We have a 6.01 SP2 IS environment (60+ production servers) and one production 6.01 Broker.

We installed a new 6.5.2 Broker and upgraded three of the minor role ISs to 6.5 SP2. Everything worked okay, though we did get some odd errors around the Broker decoder when the newly upgraded ISs connected for the first time. Synchronization resolved this error only temporarily - it appears the Broker and the Dispatcher can’t seem to agree on the document version, even if it has not changed.

Several weekends ago, we took three IS servers that represent a major business initiative and upgraded them to 6.5 SP2 and connected them to the 6.5.2 Broker. Everything seemed to be okay, until Monday rolled around and we placed a load on the Broker (normal load is around 1,000 <5K documents per minute - the peak load is around 20,000 <5K documents).

This load, by the way, is tolerated by the 6.01 Broker with no problems.

As a load was introduced, the Broker CPU utilization went really high (jumped from 1.8% to 20+%) and our IS dispatchers start throwing timeout and disconnect errors.

Our 6.5.2 Broker has sufficient stor files and a good sized RAM cache. We didn’t see any errors in the syslog or Broker logs - it just didn’t handle the document load. webMethods support hasn’t really had any ideas for us, they want us to trace and debug, etc - but of course we had to roll back ASAP for the business and have not been able to duplicate this problem in test (mainly because we can’t successfully duplicate the load.)

So we’ve all been asking around, and one of the guys heard that Brokers above 6.5 SP1 have issues. I’d find that hard to believe, but I know software companies sometimes release version they wish they had not (such as the 6.01 IS version.)

Has anyone had these kind of issues with 6.5.2 or Brokers above 6.5 SP1?

2

Phil,

We haven’t seen the type of issues you are experiencing (CPU load spiking, dispatcher issues, etc.) although we are running on AIX and are still in System Test, so not under production load.

We did struggle to get the ulimit settings right for AIX and do still see memory slowly increasing each day even under light or no load.

Our ulimit settings for Broker 6.5.2 on AIX 5.3 are:

time(seconds)        unlimited
file(blocks)         unlimited
data(kbytes)         unlimited
stack(kbytes)        1024000
memory(kbytes)       unlimited
coredump(blocks)     unlimited
nofiles(descriptors) 8192

Mark

3

Just checked the [Solaris 8] ulimits - our are set to:

time(seconds) unlimited
file(blocks) unlimited
data(kbytes) unlimited
stack(kbytes) 8192
coredump(blocks) unlimited
nofiles(descriptors) 8192
memory(kbytes) unlimited

4

Forgot to add - we think we have devised a true load test for our test system. I’ll report back our results and anything we try to resolve this.

Another thing - this 6.5.2 Broker is in a territory with a 6.01 Broker, which is the gateway broker to a 5.0 territory. webMethods support engineers tell us this is not the cause of our problems, but I’m just throwing that out there as well…

5

What’s the doc type sharing setup with the 6.01 and 5.0 Brokers? I’m wondering if the event definitions are being battled out between the different Brokers and their clients. Particularly the older Broker clients that push their event type defs on each start up. Just a random thought…

6

The 5.0 Broker can subscribe to only 2 doctypes (no filters) and publish only 3 doctypes through the gateway. Everything shows In Sync.

7

Phil, what OS are you using?

M

8

Solaris 8. And the list of important things I forgot to put in the original post grows longer and longer. :slight_smile:

Specifically, Solaris 8 2/02 s28s_u7wos_08a SPARC.

9

And to anticipate the next question, isainfo -v shows we can run both 32-bit and 64-bit applications, so the 64-bit Broker 6.5.2 is supported.

10

Are those event types the ones that suddenly go out of sync? Or is it different types?

11

No - the doctypes we have going out of sync are different than the gateway event types.

We’ve actually narrowed it down to only one doctype that continues to have the sync problem - the others seem to have resolved themselves (through our efforts or just automagically - we don’t know.)

12

Hmmm. That’s got to be aggravating. I assume you’ve gone through the painful process of verifying the 60+ servers and haven’t found anything. Ugh.

I know it’s not an easy task, but I wish wM could unify the doc type defs of IS and Broker. Having the two separate definition repositories is risky for deployments and upgrades.

13

So we’ve put a load on the test 6.5.2 Broker (70,000+ <5K documents in about 15 minutes) and were able to watch the CPU load for the Broker peak at 20%, then stabilize at 15% and merrily process documents until done.

We did not get any Endpoint unavailables or IS disconnects during that time. All other things being equal between production and test, we can only guess that it has to do with production OS settings or the physical hardware. We are upgrading the hardware this weekend, and the broker storage files location will be on fiber attached storage, rather than local write-through disk, so we hope this will resolve the problem.

Thanks to all who weighed in.

14

Wm6.5 BrokerAdmin page doesn’t appear

in wmBrokerAdmin page,when i check the status of broker it is showing as connected.
but in packages WmBrokerAdmin package is not present.
Note:while installation I had selected all the options related to broker.

15

WmBrokerAdmin is now a portlet accessible through the My webMethods console (in version 6.5+). The WmBrokerAdmin package is still available from Advantage, I think, but requires the BrokerAPIFix1 to work.

16

Well, administration functionality for Broker and BrokerJMS are now available in My webMehtods Server’s Messaging portlet, but it is a complete rewrite of the old WmBrokerAdmin and WmJMSAdmin DSP applications.

WmBrokerAdmin can no longer be downloaded using Installer if you choose 6.5.x as the Release after the “Select the webMethods release you want to install:” prompt. I just confirmed that you can still download WmBrokerAdmin 6.1 if you select “6.1.x-6.2.x” as the Release.

Also the name of the fix to Broker 6.5.2 to allow WmBrokerAdmin to connect successfully to 6.5.2 brokers is “webMethods Broker Java API 6.5.2 Fix 1”. You can find the fix on Advantage.

Mark