webMethods not able to process huge volume of documents from subscriber trigger queue

, ,
1

Hello Experts,

The current webMethods version is : 9.12 trail version.
JDBC Adapter Version: 9.10.0.0
Messaging : UM

webMethods polling 10000 records at a time from Oracle database using JDBC adapter notifications/polling notifications and inserting into other externalDB at subscriber side.
The trigger queue capacity level and refil level set as default 10 & 4 respectively. Acknowledgement Queue size is 1.

The records are inserting smoothely for sometime. After that, the webMethods IS is hunging and not at all responsive.
In addition to that we’re experiencing “java.lang.OutOfMemoryError: Java heap space” error couple of times.

  1. So, in this case how to handle. What should be the best practice for configuring the trigger queue capacity level and refil level.
    Please also let me know how to clear the trigger queues.
  2. Where do I need to increase java heap memory size in 9.12. I don’t see JAVA_MIN_MEM & JAVA_MAX_MEM in 9.12 server.bat file.

Could you please help on this issue.

Thanks,
Pradeep.

2

Hi Sai,

the Memory Settings are defined in setenv.bat under IntegrationServer/bin.
Additionally these settings are defined in profiles/IS/confiiguration/custom_wrapper.conf.

When running IS as a service you need to update the service definition in the registry and restart the service.

As long as the box has enough physical memory you can specify any value you want.

You should ensure that UM has enough space to hold all the documents.

Is this a serial or a concurrent Trigger? How many working threads are configured (only possible when using concurrent mode)?

Please check if applying IS_9.12_Core_Fix3 and JDBC_9.10_FIx2 might be neccessary.
There have been some issues with enabling polling notifications according to the readme of JDBC_9.10_Fix2.

Regards,
Holger

3

Hello Holger,

Thank you so much for your quick reply.

  1. I can see the java heap memory properties in profiles/IS/confiiguration/custom_wrapper.conf.
    I increased the heap memory as follows.
    wrapper.java.initmemory=1024
    wrapper.java.maxmemory=2048

  2. I’m running my IS as an application only.

  3. I’ve checked the IS statistics, it’s having enough physical memory more than 45%.

  4. How do I know that UM has enough space whether it can hold all the documents ?

  5. I’m currently using 9.12 trail. So, for me no fix is available. But, as I see adapter notifications working fine. The problem is with managing the documents inflow and outflow through trigger queue. And tuning the System Threads.

  6. This is a concurrent trigger.
    My current System Thread Pool settings were as follows:
    Available Threads: 78 % (78 Threads)
    Maximum Threads: 100
    Minimum Threads: 10
    Available Threads Warning Threshold : 15 % (15 Threads)
    Scheduler Thread Throttle : 100 % (100 Threads)
    Scheduler Current Threads : 0

IS>Settings>Messaging>webMethods Messaging Trigger Management

Global webMethods Messaging Trigger Controls:
Current Threads: 10
Maximum Threads : 100 (100% of Server Thread Pool)
Queue Capacity Throttle : 100%
Execution Threads Throttle : 100%

Document Retrieval: For one of the trigger service
Queue Capacity Level : 100
Document Processing :
Processing Mode : Concurrent
Maximum Threads : 1

Note: The System thread count is keep on increasing and killing the memory. For each and every poll there are around 400 system threads are creating, this count is keep on increasing with respctive to that poll. So, this leads to server shut down.

I pulled the JVM Thread Dump. There are many threads being in TIMED_WAITING state. I pulled the thread dump and tried kill these threads, but unfortunately these threads “can kill” is not supported.
In this situation how do I handle.

Please help…!

Thanks,
Pradeep.

4

Hi,

I don’t know if it’s available from the trial installation, but you should use the UM’s enterprise manager tool to check all the UM server’s stats (see IS_DIR/UniversalMessaging\java\umserver\bin\nenterprisemgr.exe).

Additional, install Oracle’s JDK and use its Java Mission Control to monitor the IS and UM JVMs (check their JMX configuration options).

Btw, the IS’s heap size is controlled by the java setting (in IS_DIR/profiles/IS_nameOfIs/configuration/custom_wrapper.conf) wrapper.java.additional.SOME_NUMBER=-XssYOUR_SIZE (as suggested in KB1765017, but you may not be able to see this note).

See “Passing Java System Properties to Integration Server” in the “webMethods Integration Server Administrator’s Guide”.

Also see: [url]java - JVM option -Xss - What does it do exactly? - Stack Overflow

Best Regards,

5

Hi,

-Xss is the paramter to control the Stack Size, Heap Size is controlled by -Xms (minimum) and -Xmx (maximum).

When running as an application modifying these paraameters and restarting the instance is sufficient.

About the increasing number of threads:
This sounds very strange to me.
Might be a bug in the JVM.

What is your exact JVM version?
Are you using the built-on version provided by SAG Installation?

You can either check for the CJP-Fixes (Java Package) for your version or you can try to install a native JDK from Oracle and use this one as an alternative.

Regards,
Holger

6

Hello Gerardo,

Yes, I don’t see any property with “wrapper.java.additional.SOME_NUMBER=-XssYOUR_SIZE” in custom_wrapper.conf.

I could see in UM Enterprise Manager as it got sufficient memory to handle these bulk of documents.

Hello Holger,
I’m using the default JVM provided with SAG 9.12 Trail version.

I could get the CJP-fixes to the current version given with SAG9.12.
But, below is the JVM version provided with SAG9.12:
java version “1.8.0_101”
Java™ SE Runtime Environment (build 1.8.0_101-b13)
Java HotSpot™ 64-Bit Server VM (build 25.101-b13, mixed mode)

As, I see still the threads are keep on creating continuously for each and every time JDBC polls, this leads to server hung & shutting down.

I’m trying to give Oracle JDK path to IS and monitor the JVM.
Could you please give me some light, in which config file I need make change Oacle JDK path as, my IS point out to Oracle JDK.

Thank you so much for all your inputs.

Thanks,
Pradeep.

7

Hi Sai,

the path to the JDK/JVM can be found in setenv.bat under IntegrationServer/instances//bin and in the profiles/IS_/config/custom_wrapper.conf.

In setenv.bat it is the variable JAVA_DIR.
In custom_wrapper.conf it is the parameter named wrapper.java.command.

Remember to restart IntegrationServer after making the changes.

The java parameter -Xss is usually not defined by default, but needs to be added when neccessary.

Regards,
Holger

8

Hi,

Sorry, I’ve made a mistake, @Holger, you are completely right.

@Sai, Have you been able to use the “Java Mission Control” and connect to the IS’s JMX port?

From your description “I see still the threads are keep on creating continuously for each and every time JDBC polls, this leads to server hung & shutting down” it seems that either the processing threads are not fast enough or they are not closing/finishing correctly.

In either case, I would advise you in reducing the processing queue size, so the IS is handling LESS work.

Increasing the notification schedule is also advised as this means LESS batches are being created.

You should then notice an increase in the UM’s load.

Also, have you tried in increasing the IS resources maximum thread number?

Did you ever got any alerts on the IS regarding available threads or memory?

The jconsole or the “Java Mission Control” might give you more information.

Best Regards,

9

Hello Holger/Gerardo,

As, of now I’m not making any changes on changing my JVM points to Oracle SDK.
Because, we’ve to demostrate to my client. If the server is not up after making the changes it would be a big headache for us.

I’ll probably make the changes once we’re done with the final demonstration is done.

Till then I’ll manage the server up when it’s not responding.

@Gerardo, The trigger queue capacity size is “100”.
As, I’m using UM as messaging component the refill size is left with default value 4 only. Because UM doesn’t process documents based on Refil level.

Also, I observed that none of the threads are not killing, even after I’ve disabled all the polling notifications & Schedulers since more than 30 hours.

The polling interval is 10 seconds we have given. As I mentioned in the previous messages below are my thread pool settings.
My current System Thread Pool settings were as follows:
Available Threads: 78 % (78 Threads)
Maximum Threads: 100 (I increased Max thread count to 100, default is 75)
Minimum Threads: 10
Available Threads Warning Threshold : 15 % (15 Threads)
Scheduler Thread Throttle : 100 % (100 Threads)
Scheduler Current Threads : 0

I didn’t receive any alerts regarding the available threads or memory.

As you said, I try Java Mission Control to know which service is causing to make the threads in TIMED_WAITING state.

Can anyone give me that java parameter -Xss configuration property. I’ll try add that in my custom_wrapper.conf for time being to see whether it’s control the stack size.

Thanks,
Pradeep.

10

You don’t need to change IS’s JVM to use the Java Mission Control, you just need to enable its JMX port, restart the IS and connect the JMC to it.

Best Regards

1 Like