Unable to create new native thread - java.lang.OutOfMemoryError

, ,
1

Hello,

I have developed a code to check the health of multiple IS (40 servers) from a single IS via remote invoke.
All the servers to be monitored are defined as ‘Remote Servers’ on the monitoring IS.

Monitoring is defined for below on the monitoring IS,

Monitors all Integration Servers status
Monitors all JDBC, SAP & MQ adapter connection status
Monitors all JDBC, SAP & MQ listener status
Monitors the queue length of all clients
Monitors the connectivity to Broker from IS
Monitors all broker servers
Monitors all file polling ports
Monitors both Memory & Thread usage on all Integration Servers
Schedule tasks on all Integration Servers
Monitors Broker/Local Triggers on all Integration Servers
Monitors the trigger throttle settings on all Integration Servers

All these monitoring activities run fine individually.
The problem arises when these are scheduled to run at specific intervals.

The schedule tasks run fine for couple of hours and then end up in error as below.
java.lang.OutOfMemoryError:unable to create new native thread

Memory initially assigned to the monitoring IS was 256.
As I encountered this error, I changed the memory to 1024 by changing the server.bat as below.
But still the error was the same. After sometime IS does not respond and finally shuts down.

set JAVA_MIN_MEM=1024M
set JAVA_MAX_MEM=1024M
set JAVA_MAX_PERM_SIZE=128M

Thread usage was well within the limit during this process.

If anyone has encountered this type of error, could you please share your thoughts.

Below are additional details.

IS version:
7.1.2.0

Extended Settings:
watt.server.cronMaxThreads=5
watt.server.cronMinThreads=2
watt.server.scheduler.maxWait=60
watt.server.scheduler.threadThrottle=75
watt.server.threadPool=75
watt.server.threadPoolMin=10

System:
Microsoft Windows Server 2003 R2

2

For me its seems to be a very good utility service which can be used in live environment. But I think few of this features is already present in Command Central in higher versions of wM 9.X

Can you try the below setting increasing the memory for 1G to 2G and re-test your all scenarios.

set JAVA_MIN_MEM=2048M
set JAVA_MAX_MEM=2048M
set JAVA_MAX_PERM_SIZE=128M

Make sure you follow the correct procedure in changing this values. For more information you can refer IS admin guide.

3

what is maximum thread value under Resources screen in IS admin portal

4

Is that error occurring very frequently. Which version of wM are you using? There are lot of threads on this error, do analysis before taking any action. Try to do the changes to memory settings in Dev before doing changes in other higher environments.

Thanks,

5

The biggest clue is in the error message itself:

java.lang.OutOfMemoryError:unable to create new native thread.

This implies that you’re not necessarily running out of heap space. You’re running out of physical memory in the server. This error is not uncommon when using native libraries or when issuing OS commands from the JVM.

Percio

6

Hi Percio,

Thanks for your comments.

Any suggestions how to handle this situation?

Thanks,
Vijay.

7

Vijay,

Sorry for not having responded more quickly. I’ve been busy and away from the forums for the last several weeks.

Have you resolved your issue? I hope so. If not, the way to address something like this is to first understand what is going on by monitoring the server closely.

I would start monitoring the IS heap usage, server’s physical memory, and server’s native processes to understand exactly what is going on. It would also be beneficial to regularly collect diagnostic data from the Integration Server via the built-in service wm.server.admin:getDiagnosticData. The output of this service contains many important tidbits, including a thread dump, which may be very revealing in your case.

Anyway, let me know if you’re still having this issue, and if so, I’ll try to help you pinpoint the root cause.

Percio