Error: java.lang.OutOfMemoryError - unable to create new native thread

,
1

Hello,

I have developed a code to check the health of multiple IS from a single IS via remote invoke.
All the servers to be monitored are defined as ‘Remote Servers’ on the monitoring IS.

Monitoring is defined for below on the monitoring IS,

Monitors all Integration Servers status
Monitors all JDBC, SAP & MQ adapter connection status
Monitors all JDBC, SAP & MQ listener status
Monitors the queue length of all clients
Monitors the connectivity to Broker from IS
Monitors all broker servers
Monitors all file polling ports
Monitors both Memory & Thread usage on all Integration Servers
Schedule tasks on all Integration Servers
Monitors Broker/Local Triggers on all Integration Servers
Monitors the trigger throttle settings on all Integration Servers

All these monitoring activities run fine individually.
The problem arises when these are scheduled to run at specific intervals.

The schedule tasks run fine for couple of hours and then end up in error as below.
java.lang.OutOfMemoryError:unable to create new native thread

Memory initially assigned to the monitoring IS was 256.
As I encountered this error, I changed the memory to 1024 by changing the server.bat as below. But still the error was the same. After some time IS does not respond and shuts down.

set JAVA_MIN_MEM=1024M
set JAVA_MAX_MEM=1024M
set JAVA_MAX_PERM_SIZE=128M

Thread usage was well within the limit during this process.

If anyone has encountered this type of error, could you please share your thoughts.

Below are additional details.

IS version:
7.1.2.0

Extended Settings:
watt.server.cronMaxThreads=5
watt.server.cronMinThreads=2
watt.server.scheduler.maxWait=60
watt.server.scheduler.threadThrottle=75
watt.server.threadPool=75
watt.server.threadPoolMin=10

System:
Microsoft Windows Server 2003 R2

Thanks,
Vijay

2

Please try set this way it depends on how much memory allocated on your OS and so IS can use it.

set JAVA_MIN_MEM=1024M
set JAVA_MAX_MEM=2028M
set JAVA_MAX_PERM_SIZE=128M

Also incresase your Resource Settings:

Server Thread Pool (just to start with)
Maximum Threads 500
Minimum Threads 100

Resart your IS

Please monitor pings and check your IS logs/Service usage.

HTH,
RMG

3

In addition to my earlier post in your duplicate post, also make sure you are on the latest fix levels of wM version 7.

I am unable to update the same message their due to the website technical issues.

4

Thank you for you inputs.

I did not increase the memory but increased the number of threads (min 100 and max 500)

And when debugged, I noticed both memory and threads are not the reason for the error.

Below are the debug logs.

2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Available Memory 90 % (932440 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:07:00 CEST [ISP.0061.0003I] Established new remote connection to (Remote Server Name) for user Administrator
2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Ping OK

2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Available Memory 90 % (932372 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:07:00 CEST [ISP.0061.0003I] Established new remote connection to (Remote Server Name) for user Administrator
2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Ping OK

2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Available Memory 90 % (932126 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:07:00 CEST [ISP.0061.0003I] Established new remote connection to (Remote Server Name) for user Administrator
2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Ping OK

2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Available Memory 90 % (931840 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Ping not OK - unable to create new native thread

2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Available Memory 89 % (930473 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:07:00 CEST [ISP.0090.0004C] Monitor – Ping not OK - unable to create new native thread

2015-07-22 14:15:02 CEST [ISP.0090.0004C] Monitor – Available Memory 26 % (275628 KB) ; Available Threads 99 % (496 Threads)
2015-07-22 14:15:02 CEST [ISP.0090.0004C] Monitor – Ping not OK - unable to create new native thread

Looking at these logs, it seems that even though there is enough memory and threads, IS is not able to obtain a connection to the remote server.

Not sure why this is happening. Please share your comments.

5

While invoking a remote service make sure you drop all the input variables not required in the pipeline this will avoid loading up all the pipeline into memory for successive calls and also check vice versa for output variables you are receiving by remote call.
You can also restrict the scope of the pipeline variables by passing only the required ones.

And also check the Keep Alive Timeout and Max Keep Alive Connections settings in remote server configurations.Check if all the remote alias connections are cleaned up properly or if there are any IS threads running for all the remote server aliases created(ping threads).

6

Yes you got to check these timeout points and make sure remote server is capable of handing it.

Assuming you have tested the remote server aliases in the sender IS.

HTH,
RMG