HTTP Post to a Delivery URL Hanging

,
1

Hi All,

In our Prod environment, post to a delivery URL using service “pub.client:http” is going to hang state and remains in hang state till we restart the server.
We have “watt.net.timeout=600” property set, which should timeout the hang threads after 600 seconds but it’s not doing so. As a result the hung threads keeps on increasing.
If anyone faced similar issues?
It would be great if anyone can suggest any solution or workaround to kill the hung threads without restarts?

Thanks!
Subha

2

Hi Subha,

did you check the “Service Usgae” page in IS Admin UI?

You should select the checkbox “show running services on top”.
Select the hanging service and stop it.

Addtionally you should check for the root cause of this hanging, i.e. why the post url does not receive a reply.

Regards,
Holger

3

in your pub.client:http service call, check if their is a value assigned for parameter timeout. it will overwrite the server’s setting for each outbound connections.

4

Thank you Holger and Tong for your reply.

Service Usage shows like pub.client:http (24) hung threads. But we are not able to kill these hung threads. SAG’s suggestion is to restart. But restart can’t be always an option. We need to know why watt.net.timeout is not terminating the connection.
No timeout values are passed in service for pub.client:http. So watt.net.timeout=600 property should be honored. But it is not.

5

Any more suggestions? Still the issue exists with pub.client:http hung threads. watt.net.timeout=600 is not honoured.

6

Hi,

Is this issue happening only for that particular HTTP Delivery URL?

As holger mentioned some is still holding the IS connection and thus the timeout setting is not honored.

What is the IS verion and fix levels in place?

Can you please have your wM admin and network teams to perform RCA?

Also what are other params you have configured under extended settings for the timeout’s in particular?

We already know the restart will definetely release the hung state threads but this is a temporary solution.

Hope you will find the resolution soon!

HTH,
RMG

7

Issue is for multiple partners/URLs.
IS version is wM 9.12 and we are in IS_9.12_Core_Fix8, Java Version:1.8.0_121 (52.0)

We noted there were a couple of threads that were blocked on a socket read in the process of sending an HTTP Post request. And in both those cases the thread dump indicated the threads were in the middle of an SSL Handshake with the remote server i.e. the thread is yet to begin sending the request to the other server.

IS Hung pub.client:http Thread shows:

Integration Server thread dump Java HotSpot™ 64-Bit Server VM (25.121-b13 mixed mode):
Thu Jan 18 07:13:18 PST 2018

“Service Thread Pool” Id=1384 in RUNNABLE (running in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.read(InputRecord.java:503)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at com.wm.ext.jsse.JSSESecureSocketFactory.newSocket(JSSESecureSocketFactory.java:567)

Not sure if this is related to java version. As per the below article java 1.8 has some bugs.

https://bugs.java.com/view_bug.do?bug_id=8075484

Any views on this?

thanks!

8

It could be …but did you get any advise from SAG support team on this issue as it clearly shows in the logs the thread got hung due to target URI was not responding in time and hence should time out and release the hung thread.?

HTH,
RMG

9

We are following up with SAG. Yet to get a proper update.

10

OK please keep this thread posted as you are dealing with 912 stable version:

HTH,
RMG

11

This issue is fixed with IS_9.12_Core_Fix13.