UM shutdown suddenly, wrraper ping jvm

Hi community,
i have a really big problem, i sould deploy some developpement but they don’t want saying that our products aren’t stable, and i need strongly your help,
The problem is that after every weekend we found our UMs down, when we check our log we found those message :

2016/10/10 11:03:47 | Shutting down Nirvana Realm
INFO | jvm 1 | 2016/10/10 11:03:48 | Realm shutdown complete
INFO | jvm 1 | 2016/10/10 11:03:48 | Nirvana Realm has been shutdown
STATUS | jvm 1 | 2016/10/10 11:03:48 | Stopping Wrapper monitor on port:9998
INFO | jvm 1 | 2016/10/10 11:03:48 | [Mon Oct 10 11:03:48 GMT 2016],Server shutdown initiated due to JVM Exit Handler called
INFO | jvm 1 | 2016/10/10 11:03:49 | [Mon Oct 10 11:03:48 GMT 2016],Shutdown: Status updates shutting down
STATUS | jvm 1 | 2016/10/10 11:03:49 | Nirvana Realm already shutdown
STATUS | wrapper | 2016/10/10 11:03:50 | ← Wrapper Stopped
STATUS | wrapper | 2016/10/10 11:39:51 | Pinging the JVM took 78633 seconds to respond.
STATUS | wrapper | 2016/10/10 13:27:27 | TERM trapped, but ignored.
STATUS | wrapper | 2016/10/10 13:34:06 | → Wrapper Sta

2016/10/15 00:46:45 | Pinging the JVM took 21 seconds

2016/10/15 03:03:54 | Nirvana Realm Server shutting down due to OutOfMemoryException in Scheduler Worker Pool:4
INFO | jvm 1 | 2016/10/15 03:05:07 | Nirvana Realm Server shutting down due to OutOfMemoryException in Scheduler
the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:16:41 | Pinging the JVM took 3 seconds to respond.
STATUS | wrapper | 2016/10/19 06:17:42 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:18:42 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:21:52 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:25:00 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:29:12 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:31:15 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:33:22 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:34:56 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:38:05 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:41:43 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:42:45 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:45:54 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:49:03 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:50:03 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 06:54:15 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 06:55:16 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:01:35 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 07:04:43 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:08:54 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:11:01 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:16:13 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:20:27 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:22:34 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:23:37 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:24:38 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:26:46 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 07:29:54 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:30:56 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:31:58 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:34:05 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:37:15 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 07:39:21 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:42:31 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 07:45:39 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:47:43 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:48:44 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:49:46 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:52:55 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:58:07 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 07:59:09 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:01:17 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 08:01:49 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:03:23 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:04:25 | Pinging the JVM took 2 seconds to respond.
STATUS | wrapper | 2016/10/19 08:07:34 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:10:39 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:13:48 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:18:26 | Pinging the JVM took 1 seconds to respond.
STATUS | wrapper | 2016/10/19 08:26:17 | Pinging the JVM took 3 seconds to respond.
STATUS | wrapper | 2016/10/19 08:27:17 | Pinging the JVM took 2 seconds to respond.

it still ping until it shutdown on the weekend,

we are on Webmethods 9,7 AIX, IS_9.7_Core_Fix9,
i found on the internet this Java Service Wrapper - wrapper.ping.alert.<x> Property but i didn’t understand what’s the relation between wrraper monitoring and UM,

any suggestions please,

Regards,
Nezha

Hi Nezha,

It seems to be issue with Java Heap Memory.

What is current heap that you have configured for UM?

Please check and let me know. (location: \SoftwareAG\UniversalMessaging\server\umserver\bin\Server_Common.conf)]

Also let me know the value of “Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

Regards,
Krishna

Can you please provide the UM fix level?

Try to apply the latest fix , hope this will resolve the issue.

Regards,
Krishna

hi Krishna PB ,
thank you for your reply,
the heap jvm configured on Server_Common.conf is 2024,

about Event Storage → QueueDeliveryPersistentPolicy could you please tell me how to verify this??
our fixes ares :
Universal Messaging Java Client 9.7 SP0 Fix 3
Universal Messaging Enterprise Manager 9.7 SP0 Fix 3
Universal Messaging Realm Server 9.7 SP0 Fix 3

Regards,
Nezha

Hi Krishna,
another setting to check is Event Storage > EnableStoreCaching. It should be set to false.
You can check/change this setting (and the one metioned by Nezha) in Enterprise Manager. Click on the Realm, select the Config tab and then find the right setting. Double-click it to change the value.

One other possible cause may be queues or channels without active subscribers. Can you look on the disk in /UniversalMessaging/server//data?
Are there any very large files in this directory or any of its subdirectories?

Hi Jonathan,

I think you misunderstood , actually Nezha is facing the issue and I suggested to check her.

Nezha,

Follow the below instruction to resolve the issue.
- Try to update the UM fix level to 9.7.0.6/latest. by using update manager
- Change the setting as I mentioned in my previous post.
“Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

To check and change the value for QueueDeliveryPersistentPolicy follow the below steps…

  - login to the Enterprise Manager 
  - Click on  realms 
  - Go to the Config tab
  -Go tot the Event Storage , there you can find the above field and see the value and change it accordingly.
  - Have attached the screen shot for your reference.
  • Also check once what Jonathan has mentioned in previous post.

Please let me know in case of any concerns.

Regards,
Krishna

hi Krishna and Jonathan ,
thank you all for your reply,
@Krishna i will apply the solution that you propose and come back to you for the result,
i want just to tell @Jonathan that Under this directory /UniversalMessaging/server//data we have very large files which is nirvana logs,

Regards,
Nezha

hi,
however on this directory /UniversalMessaging/server//bin there is a lot of very large files dmp & phd & javacore****.txt & .trc,

Regards,
Nezha

Nezha,
log files are quite normal and you can safely delete old ones if you want to recover disk space.
The files in the bin directory look like heap dumps which may have been created automatically by the JVM if it crashed. You can safely delete dmp, phd trc and javacore*.txt files if you don’t think you need them for analyzing some failure.

hi,
could you please tell me which one of thos fixes should i install, see attchement file,

Regards,

Hi Nezha,

I think you should at least apply Fixes for EnterpriseManager, RealmServer and SPM.

Remember that all UM Components working with each other should have the same fix level.

Regards,
Holger

hi everyone,
Even after installing last fixes and doing all recomended changes, wrraper still pinging the JVM :frowning: see attached files,

is there anything to do?

because it beging with pinging jvm and it shuttdown after!!

we have 3 environnement DEV REC and PROD?

we havn’t any service or flux runed on REC and PROD environnement, we are in the bigining we still developping on DEV environnement,

in DEV wrraper ping JVM always(i made changes on this environnement) , on REC it ping sometimes and it shuttdown on weekend, on PROD there is no problem untill now,

Regards,
Nezha

why they are shutting down only on weekends … Any specific reason ? Did you ask your OS admin team to know any daemon services running which are stopping your UM services … ?

Thanks,

Hi,

this might be a license issue.

When running with specific (invalid) type of license it might force the server to shutdown after a certain period.

Regards,
Holger