Enterprise Gateway (Standard gateway) throwing 404

, ,
1

Hello Everyone,

Am facing a very strange issue. The enterprise gateway (Standard Gateway) throwing 404 randomly for all apis.
The behavior is very strange sometime request are being processed successfully but in the next moment Standard gateway throw error 404 not found i have verified that the the service and URL are available below is the complete message.

{
    "Exception": "API Gateway Stander: Service not found: /abc/xyz/v1.0/ . Request Details: Service - null, Operation - null, Invocation Time:7:50:42 PM, Date:Sep 2, 2024,  Client IP - 111.00.xx, User - Default and Application:null -------------------------------null"
}

I checked the logs for the standard gateway the last logs I found in server are below.

2024-09-03 11:16:24 AST [ISS.0053.0009W] (tid=16574) (tid=1072417505) Enterprise Gateway Server encountered an I/O error while writing the Internal Server's response back to the client at "100.2.2.2". The request URL is "/gateway/external/xyz/"
2024-09-03 12:14:55 AST [ISS.0053.0009W] (tid=16574) (tid=1072417505) Enterprise Gateway Server encountered an I/O error while writing the Internal Server's response back to the client at "100.2.0.0". The request URL is "/gateway/external/abc".
2024-09-03 12:16:04 AST [ISS.0053.0009W] (tid=18504) (tid=631908035) Enterprise Gateway Server encountered an I/O error while writing the Internal Server's response back to the client at "100.2.0.0". The request URL is "/gateway/external/all".

2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist

[s-mhaseeb@ryd-ssa-nft-apigws-01 logs]$ tail -f server.log
2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:31:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:18 AST [YAI.0103.0120D] (tid=600) [default][ryd-ssa-nft-apigws-01]  [Find Records] Total number of records in response List for Index name - [gateway_default], type - [microGatewayRegistrationInfo] is  0
2024-09-03 13:32:23 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:23 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:23 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:25 AST [YAI.0006.8887D] (tid=547) [default][ryd-ssa-nft-apigws-01]  Debug: read all available packages
2024-09-03 13:32:25 AST [YAI.0005.8887D] (tid=547) [default][ryd-ssa-nft-apigws-01]  Debug: isTeamWorkEnabled result : false
2024-09-03 13:32:25 AST [YAI.0103.0120D] (tid=547) [default][ryd-ssa-nft-apigws-01]  [Find Records] Total number of records in response List for Index name - [gateway_default], type - [packages] is  0
2024-09-03 13:32:25 AST [YAI.0005.8887D] (tid=547) [default][ryd-ssa-nft-apigws-01]  Debug: isTeamWorkEnabled result : false
2024-09-03 13:32:25 AST [YAI.0206.8887D] (tid=547) [default][ryd-ssa-nft-apigws-01]  Debug: starting stale quota remove process
2024-09-03 13:32:25 AST [YAI.0103.0120D] (tid=547) [default][ryd-ssa-nft-apigws-01]  [Find Records] Total number of records in response List for Index name - [gateway_default], type - [quotaaccumulator] is  0
2024-09-03 13:32:25 AST [YAI.0206.8887D] (tid=547) [default][ryd-ssa-nft-apigws-01]  Debug: finished stale quota remove process
2024-09-03 13:32:33 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:33 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:33 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:43 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:43 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:43 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:32:48 AST [YAI.0103.0120D] (tid=600) [default][ryd-ssa-nft-apigws-01]  [Find Records] Total number of records in response List for Index name - [gateway_default], type - [microGatewayRegistrationInfo] is  0
2024-09-03 13:32:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:32:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:32:53 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:33:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:33:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:33:03 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:33:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting Event in batch as time interval reached
2024-09-03 13:33:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: Persisting events to Gateway as batch. batch count - 0
2024-09-03 13:33:13 AST [YAI.0300.8887D] (tid=541) [default][ryd-ssa-nft-apigws-01]  Debug: No Request to persist
2024-09-03 13:33:18 AST [YAI.0103.0120D] (tid=600) [default][ryd-ssa-nft-apigws-01]  [Find Records] Tot

Thanks

2

Can you share the details of your setup? How many nodes in the cluster?

We’ve encountered similar when an IS loses connection to its ElasticSearch companion or when ES is misbehaving for some reason. If you have just 1 node, then the issue will require additional research to figure out. If you have 2 nodes, that’s a problem when one of the nodes is restarted. Refer to the docs on the web for why an ES cluster must have at least 3 nodes, never 2.

3

Hello Everyone,

The issue is resolved.

Setup Details :

We have 3 APIGW nodes for Advance(DMZ) and 2 node for Standard/Enterprise (Public).
In our setup APIGW Standard is just acting as reverse proxy, which is responsible to route the request to APIGW Advance.

Resolution Details :
In my case the default secure port was enabled and was receiving the requests and throwing 404. We have configured different port for the communication between Advance & Standard gateway.
I resolved the issue by disabling the default secure port.