We have an interesting situation that we’ve encountered that I’m hoping someone else has encountered as well and perhaps has a solution.
We have a .Net real time service (written in C#.Net 4.5) hitting a webMethods web service. This real time service runs on two difference services and hits the webMethods ESB server through a load balancer. When going through the load balancer we will intermittently get a HTTP 400 [ISC.0064.9101] Bad Request
In our network traces we can see no evidence of an issue with the request that is being rejected. We suspect this is a connection issue but we have yet to be able to see a smoking gun in any of our traces.
The request appears to be http 1.1 but I’m not sure that would cause the issue.
The real time processes go through a large sequential volume but not a large concurrent volume. So each server will only ever send one request to the ESB at a time. So it’s not a concurrent volume issue. But they are at times sending one request after another for a large volume.
We have been able to take logs but they haven’t necessarily made sense.
What we see is a pattern of our request, and a success response, and then the 400 Bad Request coming out of nowhere.
It makes me think that it’s a dropped connection. The request that fails according to the calling system and the back end results never seems to make it to web methods.
The error though is far from helpful and the traces so far have shown nothing
Our network team is doing some more analysis but I was wondering if anyone one else had encountered something similar
Not sure what you mean by TN URL? Our Flow service has a WSDL so they are hitting it through that
When a load balancer receives an HTTP request, it checks for malformed requests and for the length of the method. The total method length in an HTTP request to a load balancer must not exceed 127 characters. If the HTTP request passes both the checks, the load balancer sends the request to the back-end EC2 instance. If the method field in the request is malformed, the load balancer responds with an HTTP 400: BAD_REQUEST error.
agreed with MR as173d, Kindly increase log levels in load balancer and IS. This will help to identify, where its breaking.
Thanks Everyone, I really appreciate the responses.
We know from the network trace, and from the above, that it’s not the Load balancer that is returning the Bad Request, it’s actually webMethods. The ISC code that is returned (as per the subject line) is a webMethods error code.
We have increased the logging level and SoftwareAG support is stumped so far, which is why I’m reaching out to see if anyone else had something similar.
Our network team is still going through the latest trace, once I get those results I’ll share what I can
We are calling it form Apigee application … We’ve got multiple calls to IS for 1 flow (request + some logs). When we have turned off logs - it looks fine. Now we are investigating with SAG what was the root case. We will check the keepAlive stuff there. Thanks for pointing it out.
What port is the requesting coming to in webMethods? Check the settings on the port…especially the backlog setting. Increase this number to the max(65534) number and see if that resolves the issue. Also, increase the keep alive timeout on the port from default 20 seconds to 60 seconds.
Just wanted to write that we’ve resolved the issue. The problem was in both places: our development and how the IS interpret REST requests.
So I would expect that if something is wrongly developed - it just won’t work (or will work in the same way in all calls). But that is not true in that case - the IS interpret wrong request sometimes as fine, and sometimes has returned the error as mentioned here.
The problem was that in the new service that has been created on our side - the REST GET request sometimes has BODY and it shouldn’t.
The line that has caused the issue:
context.setVariable(“request.content”, (requestPath + queryString));
When this line has been commented - everything start to work fine.
The strangest thing was that this hasn’t cause the issues for all calls that we have done.
So be careful with copy/past implementation from other code
That is really interesting. In our case we were connecting via SOAPUI and using Microsoft’s .Net core libraries so I don’t believe for us this is the issue.
But hopefully this will help people who are having issues via REST. Thanks for responding
