Integration Server keeps restarting

Hi All,

I am new to webMethods.

My Integration server keeps restarting even after I stop the server.

My requirement is to keep the server down forever as we don’t require it for now.

But as it keeps restarting , there is extra manual work of stopping the server daily. Since this is not the case with just one server, I need to find out the reason as it will save my lot of time of daily checking on those servers and stopping them.

Could you please help me here.

Thanks for your help in Advance!

Hi Priya,

can you explain how you stop your server as there are several possibilities to do so with different outcome?
Some of them will auitomatically restart the server and some of them will not.

Is the server running as a Windows service or as an application?

Can you provide the wrapper.logs from profiles/IS_default/logs?

Which version of wM are you using?

Regards,
Holger

Interesting and What is your server version (hosted OS on) and you said shutting down manually and can you tell or do you know when that server is starting up per the IS/wrapper logs?

Please post here with more analysis details to resolve this kind of issue.

HTH,
RMG

1 Like

Hello Holger,
Thank you for coming back on this.
As I already mentioned, I wanted the server to be always down. So I am just stopping the server. It is getting started itself(dont know how).
I am stoppping the server via unix box where we first remove the LOCKFILE generated in the IntegrationServer directory. Then kill the instance running on specific port(port which needs to be down).

It gets start itself.

Runnning as an application.

Could not find any wrapper.log under profiles/IS_default/logs.
Found under /common/runtime/logs but with ‘0’ file size.

version = 8.2

Regards
Priya

Interesting and What is your server version (hosted OS on) and you said shutting down manually and can you tell or do you know when that server is starting up per the IS/wrapper logs?

Please post here with more analysis details to resolve this kind of issue.

HTH,
RMG

==================================================

Hi RMG,

I am not able to find any kind of wrapper.logs logged in the server.

server versoin = 8.2

Regards
Priya

Hi Priya,

in this case the wrapper.log is located under profiles/IS/logs.

Removing the LOCKFILE and killing the Unix process is not the recommended way to shutdown the IntegrationServer.
You should either use shutdown.sh from IntegrationServer/bin, the shutdown link in IS Admin UI or just remove the .anchor file from profiles/IS/bin.

This informs the wrapper the IntegrationServer should not be restarted after shutting down.
LOCKFILE will be automatically removed at the end of the shutdown procedure.

Wrapper will even take care of the fact that the IS could not be shutdown correctly by killing the IS Unix process after certain time.

When just killing the Unix process for the IS the wrapper assumes that the IS crashed accidentally due to some JVM error and restarts it immediately.

Regards,
Holger

1 Like

I would also take a look at your cron jobs. It’s not uncommon for people to create auxiliary scripts to monitor the Integration Server, and if the script detects that the server is down, it will automatically start it back up.

Percio

1 Like

Hi Priya,

in this case the wrapper.log is located under profiles/IS/logs.

Removing the LOCKFILE and killing the Unix process is not the recommended way to shutdown the IntegrationServer.
You should either use shutdown.sh from IntegrationServer/bin, the shutdown link in IS Admin UI or just remove the .anchor file from profiles/IS/bin.

This informs the wrapper the IntegrationServer should not be restarted after shutting down.
LOCKFILE will be automatically removed at the end of the shutdown procedure.

Wrapper will even take care of the fact that the IS could not be shutdown correctly by killing the IS Unix process after certain time.

When just killing the Unix process for the IS the wrapper assumes that the IS crashed accidentally due to some JVM error and restarts it immediately.

Regards,
Holger

Hi Holger,

Appologies for the confusion here.

The procedure I mentioned is done in case of RRT servers and that also when the shutting down is not happening properly.

For prod servers I follow the one you mentioned i.e. running the shutdown.sh script under bin path.

I had checked under the path /profiles/IS/logs also, but did not find any wrapper.log being generated.

Infact I even tried to check the path configgured for wrapper.log to be generated in wrraper.conf file and that also directs the same path i.e. /profiles/IS/logs/wrapper.log . But when I got there, I dont find any such file.

Also I tried running a find cmd to check all the possible location for wrapper.log file and thats when I found it under the path common/runtime/logs/ but again when I go there find the file with name wrapper.log but with 0 byte size.

Please let me know, what am I missing.

Thank you for your response. :slight_smile:

Regards
Priya

Hello Percio,

This was the first thing I looked into :slight_smile:

There is no such script placed in the server which could be responsible for restart of the server.

Regards
Priya

Hi Priya,

do you have the nohup.out available for this IntegrationServer?
Eventually this one provides us more informations.

Are you able to identiy why the shutdowns is not happening properly?
When using the shutdown.sh from either IntegrationServer/bin, profiles/IS/bin or just deleting the anchor file under profiles/IS/bin the shutdown should be completed successfully.

Regards,
Holger

Hello Holger,

Apologies for delay in response.

Yes we do have nohup.out available for the IS. What kind of info will be helpful from this file to find out the cause of server getting restarted.

We are shutting down the server following proper process. But not able to find why it is getting restarted again.

Regards
Priya

Priya,

Since you’re not finding the wrapper.log file, I’m guessing you’re not running the server inside the Tanuki wrapper. However, just in case you are, you should know that, as Holger has indicated, the wrapper will take care of restarting the server if it thinks the server crashed.

This is typically handled/configured by the Tanuki wrapper’s on_exit property. You should be able to find this property in your wrapper.conf and custom_wrapper.conf files. Play around with these properties to see if they are the reason why your server is restarting automatically. If you find out that after tweaking these properties, the server stopped restarting automatically then one of two things:

(a) The server is still not being shutdown properly
(b) There is a bug

You can find out more about this property here: wrapper.on_exit.<n> Property - Java Service Wrapper

Percio

1 Like

Percio,

This information was really useful.

But wen I go to the path /profiles/IS/configuration/wrapper.conf and check for the property mentioned above, i.e. 'on_exit ’ ,I don see any as such. :frowning:

And custom_wrapper.conf file is empty.

Regards
Priya

Hi,

Is it different if you shut it down using:

  • web interface, “Shut Down and Restart”;
  • SAG’s shutdown script
  • calling the internal shutdown service
  • making a system shutdown of the service (if installed as a service)

Best Regards,

Hi Gerardo,

When I am trying to stop down the server using shutdown.sh script from bin path, below server.logs are been generated and also I could not see server going down. :frowning:

Please have a look:

bash-3.2$ tail -f server.log
2017-07-01 23:45:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:45:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting t
2017-07-01 23:46:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:46:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:46:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting t
2017-07-01 23:46:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:46:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:47:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting t
2017-07-01 23:47:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:47:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:47:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting t
2017-07-01 23:48:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:48:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:48:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting t
2017-07-01 23:48:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:48:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Sto
2017-07-01 23:49:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:49:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:49:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:49:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:50:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:50:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:50:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:50:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:50:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:51:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:51:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:51:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:51:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:52:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:52:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:52:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:52:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:52:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:53:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:53:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:53:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:53:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:54:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:54:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:54:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:54:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:54:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:55:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:55:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:55:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:55:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:56:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:56:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:56:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:56:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:56:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:57:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:57:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:57:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:57:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:58:09 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:58:09 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:58:39 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:58:49 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:58:49 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:59:19 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll
2017-07-01 23:59:29 BST [ISS.0025.0025I] Broker Synchronizer initialized
2017-07-01 23:59:29 BST [ISS.0098.0042I] Successfully reconnected to Broker. Stopped polling
2017-07-01 23:59:59 BST [ISS.0098.0041C] Unable to connect to Broker. Starting to poll

Regards
Priya

Hi,

In some installations the startup scripts do not work properly…

See what happens when you kill -9 the wrapper’s process or the IS process.

Until 9.5 the scripts can be broken quite easily.
Do you have another installation you can compare against?

Best Regards,

Hi Gerardo,

It gets killed. And the server becomes down.

Regards
Priya

Hi,

In certain installations, the web admin ‘shutdown’ action does not work.

I don’t know why but I suspect the script maybe in a loop.

Using the scripts to start and stop the IS usually works.

Best Regards,