UM shutdown suddenly, wrraper ping jvm

Nezha_Abrabri · October 19, 2016, 12:03pm

Hi community,
i have a really big problem, i sould deploy some developpement but they don’t want saying that our products aren’t stable, and i need strongly your help,
The problem is that after every weekend we found our UMs down, when we check our log we found those message :

2016/10/10 11:03:47 | Shutting down Nirvana Realm
INFO | jvm 1 | 2016/10/10 11:03:48 | Realm shutdown complete
INFO | jvm 1 | 2016/10/10 11:03:48 | Nirvana Realm has been shutdown
STATUS | jvm 1 | 2016/10/10 11:03:48 | Stopping Wrapper monitor on port:9998
INFO | jvm 1 | 2016/10/10 11:03:48 | [Mon Oct 10 11:03:48 GMT 2016],Server shutdown initiated due to JVM Exit Handler called
INFO | jvm 1 | 2016/10/10 11:03:49 | [Mon Oct 10 11:03:48 GMT 2016],Shutdown: Status updates shutting down
STATUS | jvm 1 | 2016/10/10 11:03:49 | Nirvana Realm already shutdown
STATUS | wrapper | 2016/10/10 11:03:50 | ← Wrapper Stopped
STATUS | wrapper | 2016/10/10 11:39:51 | Pinging the JVM took 78633 seconds to respond.
STATUS | wrapper | 2016/10/10 13:27:27 | TERM trapped, but ignored.
STATUS | wrapper | 2016/10/10 13:34:06 | → Wrapper Sta

2016/10/15 00:46:45 | Pinging the JVM took 21 seconds

it still ping until it shutdown on the weekend,

we are on webMethods 9,7 AIX, IS_9.7_Core_Fix9,
i found on the internet this [b]https://wrapper.tanukisoftware.com/doc/english/prop-ping-alert-x.html[/b] but i didn’t understand what’s the relation between wrraper monitoring and UM,

any suggestions please,

Regards,
Nezha

Krishna_PB · October 19, 2016, 4:40pm

Hi Nezha,

It seems to be issue with Java Heap Memory.

What is current heap that you have configured for UM?

Please check and let me know. (location: \SoftwareAG\UniversalMessaging\server\umserver\bin\Server_Common.conf)]

Also let me know the value of “Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

Regards,
Krishna

Krishna_PB · October 19, 2016, 4:46pm

Can you please provide the UM fix level?

Try to apply the latest fix , hope this will resolve the issue.

Regards,
Krishna

Nezha_Abrabri · October 19, 2016, 6:01pm

hi Krishna PB ,
thank you for your reply,
the heap jvm configured on Server_Common.conf is 2024,

about Event Storage → QueueDeliveryPersistentPolicy could you please tell me how to verify this??
our fixes ares :
Universal Messaging Java Client 9.7 SP0 Fix 3
Universal Messaging Enterprise Manager 9.7 SP0 Fix 3
Universal Messaging Realm Server 9.7 SP0 Fix 3

Regards,
Nezha

Jonathan_Heywood · October 19, 2016, 6:11pm

Hi Krishna,
another setting to check is Event Storage > EnableStoreCaching. It should be set to false.
You can check/change this setting (and the one metioned by Nezha) in Enterprise Manager. Click on the Realm, select the Config tab and then find the right setting. Double-click it to change the value.

One other possible cause may be queues or channels without active subscribers. Can you look on the disk in /UniversalMessaging/server//data?
Are there any very large files in this directory or any of its subdirectories?

Krishna_PB · October 20, 2016, 8:53am

Hi Jonathan,

I think you misunderstood , actually Nezha is facing the issue and I suggested to check her.

Nezha,

Follow the below instruction to resolve the issue.
- Try to update the UM fix level to 9.7.0.6/latest. by using update manager
- Change the setting as I mentioned in my previous post.
“Event Storage → QueueDeliveryPersistentPolicy”. If the default value is “NoPersist/NoSync” , then try to change to “Persist/NoSync” .

To check and change the value for QueueDeliveryPersistentPolicy follow the below steps…

  - login to the Enterprise Manager 
  - Click on  realms 
  - Go to the Config tab
  -Go tot the Event Storage , there you can find the above field and see the value and change it accordingly.
  - Have attached the screen shot for your reference.

Also check once what Jonathan has mentioned in previous post.

Please let me know in case of any concerns.

Regards,
Krishna

Nezha_Abrabri · October 20, 2016, 11:12am

hi Krishna and Jonathan ,
thank you all for your reply,
@Krishna i will apply the solution that you propose and come back to you for the result,
i want just to tell @Jonathan that Under this directory /UniversalMessaging/server//data we have very large files which is nirvana logs,

Regards,
Nezha

Nezha_Abrabri · October 20, 2016, 11:46am

hi,
however on this directory /UniversalMessaging/server//bin there is a lot of very large files dmp & phd & javacore****.txt & .trc,

Regards,
Nezha

Jonathan_Heywood · October 20, 2016, 11:50am

Nezha,
log files are quite normal and you can safely delete old ones if you want to recover disk space.
The files in the bin directory look like heap dumps which may have been created automatically by the JVM if it crashed. You can safely delete dmp, phd trc and javacore*.txt files if you don’t think you need them for analyzing some failure.

Nezha_Abrabri · October 24, 2016, 6:06pm

hi,
could you please tell me which one of thos fixes should i install, see attchement file,

Regards,

Holger_von_Thomsen · October 24, 2016, 6:27pm

Hi Nezha,

I think you should at least apply Fixes for EnterpriseManager, RealmServer and SPM.

Remember that all UM Components working with each other should have the same fix level.

Regards,
Holger

Nezha_Abrabri · October 25, 2016, 10:52am

hi everyone,
Even after installing last fixes and doing all recomended changes, wrraper still pinging the JVM see attached files,

is there anything to do?

because it beging with pinging jvm and it shuttdown after!!

we have 3 environnement DEV REC and PROD?

we havn’t any service or flux runed on REC and PROD environnement, we are in the bigining we still developping on DEV environnement,

in DEV wrraper ping JVM always(i made changes on this environnement) , on REC it ping sometimes and it shuttdown on weekend, on PROD there is no problem untill now,

Regards,
Nezha

MR_as173d · October 29, 2016, 12:47pm

why they are shutting down only on weekends … Any specific reason ? Did you ask your OS admin team to know any daemon services running which are stopping your UM services … ?

Thanks,

Holger_von_Thomsen · October 31, 2016, 5:39pm

Hi,

this might be a license issue.

When running with specific (invalid) type of license it might force the server to shutdown after a certain period.

Regards,
Holger

Topic		Replies	Views
High heap memory use webMethods , Universal-Messaging-Broker , Integration-Server-and-ESB	14	2632	April 2, 2021
can't start UM webMethods , Universal-Messaging-Broker , Integration-Server-and-ESB	6	2322	April 2, 2021
UM not coming up webMethods , Universal-Messaging-Broker , Integration-Server-and-ESB	3	1140	April 2, 2021
UM Node shutting down with error "Nirvana Realm Server shutting down due to UserManager: OutOfMemoryError raised in User" webMethods , Universal-Messaging-Broker	18	96	September 9, 2024
Error while connecting to umserver using Java Admin API webMethods , Universal-Messaging-Broker , Integration-Server-and-ESB	16	5682	April 2, 2021

UM shutdown suddenly, wrraper ping jvm

2016/10/15 00:46:45 | Pinging the JVM took 21 seconds

Related topics