I have seen many posts already existing regarding this subject, but i wanted a clear picture on it.
Scenario: I have an IS service, which had hung due to some issue(Already identified). So i need to kill that thread(Know that it shouldn’t be done often), by IS admin console. But unfortunately after cancelling, its still not getting killed.
I saw some of the java services already being developed for this, but i dont wanna use one more thread to kill a hung thread in JVM level.
Is restarting IS is the last resort for this?
Hi Niteesh,
As you said, you can ‘cancel’ the hung thread, followed by ‘kill’ option enabled.
What exactly you mean when you say, “don’t want to use one more thread to kill a hung thread in JVM level”? To kill something in JVM level, we got to issue a command in JVM level. What other option is available?
Yes, restarting IS or killing IS thread is the last option if the thread is not getting released.
Thanks Senthil for the reply. What i meant was, using java native API for thread interruption we can kill a thread. But i dont think that will work work for intrinsic threads… Never mind. Better to restart the server instead of messing with jvm.
Do you see ‘Kill’ option enabled after ‘cancel’ thread option. Can you try to ‘cancel’ thread multiple times and see if Kill is enabled or not?
This feature was introduced in order to avoid Server restart’s due to specific hung thread. What kind of service is hung? We can still check other ways before restarting the server I think.
Yes, kill option(Red cross) is enabled. After clicking(repeatatively) also, its not getting killed. Service is having a repeat step which is executing infinitely.
I just tried with a simple flow service that executes a repeat in an infinite manner. Soon after cancelling the running thread from IS admin, I see below message and execution stopped.
Could not run ‘runInfinite’.
com.wm.lang.flow.FlowException: [ISC.0049.9030] User Administrator issued request to terminate the flow.
Oh… Thats great… But mine is executing ssh command inside Repeat loop. So there was a network issue and it kept on repeating. Now if i try to kill that thread, new threads are getting created which inturn trying to kill hung thread(Fascinating ).
Yes, even with the new facility. Killing a thread in a JVM is generally discouraged. Too many potential side-effects. Even the SAG documentation warns about cancelling or killing threads–“Killing a thread might put your resources in an unstable state.”
In other words, it is a roll of the dice. It may or may not successfully cancel/kill the thread. It may or may not cause additional problems. A hung thread is usually the result of not accounting for something within the code. E.g. not specifying a timeout. Or simply a bug. If one finds that they are regularly killing threads, something is wrong.
Yes, I agree. But most of the time it will work except few cases. I observed that, if it is not getting killed, than you should not try it more than once or twice and restarting is the best resort.
I tried to kill the thread using admin console twice before, but It didn’t get killed. so I went for a restart…But I sucessfully killed the thread few months back…So, we are not sure when this will work out and when it will not…Still I feels, bettter option is going for a restart.
I would advise, first check the reason for hanging(Make sure that it wont happen after you restart). Also try to kill one or two times max from admin panel. If still does not work,then go for restart and don’t try to kill it repetitively(It wont work and increase unnecessary threads).
Also make sure you take 3 Java core thread dumps during IS hung kill -3 PID (with minute a part for 3 times) and open a ticket with SAG and provide them (server,stats,error log) and you can also do some analysis from the server/error logs for more troubleshooting analysis.
).