webMethods API Gateway tutorial
Author: Chidambaram, Arun Dev (arunc@softwareag.com)
Supported Versions: 10.3 & above
Overview of the tutorial
API Gateway is dependent on various components like Integration Server,Internal Data Store, OSGI, Terracotta etc. Logs are generated by each of these components and stored in different locations. It becomes a cumbersome process to get these logs, set the log level and do the analysis when an error occurs in API Gateway. API Gateway in 10.3 simplifies this logging functionality so that customers can configure the log levels in a single place, download the logs and also enable log aggregation from different components. This tutorial will guide you on setting the log levels, downloading the logs, enabling log aggregation and also some of the best practices that should be followed.
Details
The log data is very important as it can be used in the following scenarios:
- Monitoring the run time anomalies in API Gateway
- Identifying the root cause of failures in API Gateway
- Identifying the configuration issues in API Gateway
- Detecting log patterns across the nodes in cluster
API Gateway is dependent on various components which has its own logging infrastructure.The various dependent components of API Gateway are as follows:
- Integration Server
- Internal Data Store
- Platform (WSStack, OSGI etc)
- Terracotta
API Gateway provides simplified solution with a one stop place to do the following actions:
- To set the log level
- Download the logs
- Aggregate the logs from different components.
- View the logs from different components in a single place and perform data mining
Platforms supported
This functionality is supported for both Windows and Unix platforms.
Setting the log levels for different components
The log levels for different components can be configured in the "Administration → General→ Application logs" section as shown below.
Explanation of log settings for the different components are shown below:
#
|
Log file type
|
Description
|
Available log levels
|
Source of the log file
|
---|---|---|---|---|
#
|
Log file type
|
Description
|
Available log levels
|
Source of the log file
|
1 | API Gateway |
This setting changes the log level of API Gateway server. All the design time activities like creation of API,Application,Alias etc and the run-time activities like API invocation will be logged based on this setting |
off, trace, debug, info, warn, error, fatal | <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/server.log |
2 | Internal Data Store logs |
This setting changes the log level of Internal Data Store. All API Gateway assets like API,Application,Alias,Policy etc are stored in this Internal Data Store |
off, info, debug, warn, error, fatal | <APIGatewayInstallationDir>/InternalDataStore/logs/SAG_EventDataStore.log |
3 | Dashboard |
This setting changes the log level of Dashboard. API Gateway internally uses Kibana to show graphs and visualize transnational data |
silent, quiet, verbose | <APIGatewayInstallationDir>/profiles/IS_default/apigateway/dashboard/startup.log |
4 | Security |
Enabling this logger will store the security events that occur in API Gateway |
enable or disable | <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSECURITY*.log |
5 | Session |
Enabling this logger will store the session details like session id, session state, server id, timestamp etc for every login and logout happening in API Gateway |
enable or disable | <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSESSION*.log |
6 | Platform |
This setting changes the log level of components that are available at the API Gateway platform like rampart,axis2,OSGI etc. Setting this log will require a API Gateway restart |
off, trace, debug, info, warn, error, fatal | <APIGatewayInstallationDir>/GatewayTrunk/profiles/IS_default/logs/sag-osgi.log |
Downloading diagnostics
The download diagnostic button is used to download diagnostics data for API Gateway. The diagnostics data that are collected are as follows.
- Logs from different components like API Gateway Server, API Gateway UI, Internal Data Store, Security, Session, Platform , Wrapper.
- Configurations and watt settings that are set in API Gateway
- A thread dump of all the currently active threads.
All the above data are compressed and sent back. Since the data that is being collected is huge, it takes some time to download the diagnostic data.
Log aggregation
Enabling log aggregation will collect logs from the different components and store in Elasticsearch. There are 2 options to choose on where to store the logs.
API Gateway store
API Gateway comes with an Internal Data Store which is used to store assets in API Gateway.
External Elasticsearch
The external Elasticsearch server details such as host name, port, etc should be configured.
The Elasticsearch server configuration details are given below.
S.No
|
Fields
|
Description
|
---|---|---|
1 | Protocol | The type of protocol (HTTP, HTTPS) to use for the host and port combination. |
2 | Hostname | Specifies the host name or IP address of the machine on which external Elasticsearch resides. |
3 | Port | Specifies the port where external Elasticsearch server runs. |
4 | Indexname | Specifies the indexname which will be used by the external Elasticsearch to store the aggregated logs |
5 | Username | Specifies the Elasticsearch user ID for authenticating Elasticsearch when API Gateway communicates with it. |
6 | Password | Specifies the password of the Elasticsearch instance to be used for establishing communication between API Gateway and Elasticsearch. |
Viewing the logs
Once log aggregation is enabled , API Gateway starts collecting the logs from various sources. These logs can be visualized and downloaded by navigating to "Analytics → Application logs". This analytics page will be rendered only if log aggregation destination is "API Gateway Store".Customer has to have their own tool to visualize the collected logs when "External Elasticsearch" destination is used to store aggregated logs.
Archiving, Purging and Restoring
Once the log aggregation is enabled , API Gateway starts collecting logs from various sources. These logs can be archived, purged and restored by going to "Administration → Manage data". The archive, purge and restore can be done from this screen only if the log aggregation destination is "API Gateway Store". Customer has to have their own logic for archiving, purging or restoring when "External Elasticsearch" destination is used to store aggregated logs.
Log configuration flavors
API Gateway internally uses Filebeat 6.0.1 to monitor the log files and forwards them to either API Gateway Store or external ElasticSearch. Filebeat is shipped along with API Gateway and itsconfiguration can be found in the following location "/<APIGateway Install Location>/profiles/IS_default/apigateway/filebeat/filebeat_apigateway.yml".
When log aggregation is enabled API Gateway starts Filebeat process in the background and it starts monitoring the log files and forwards them to either API Gateway Store or external Elasticsearch.
API Gateway + Internal Data Store
This is the Out of the box support available in API Gateway for log aggregation. Customers can just configure from the steps listed above and start the aggregating logs from different components.In this approach the same Internal Data Store is used for storing the API Gateway metadata like APIs, policies, configurations, run-time events etc. So if the log aggregation is also enabled then the Internal Data Store might be overloaded with too many requests and data might grow exponentially. So the administrators should keep track of the log aggregated data and periodically do purging.
API Gateway + External Elasticsearch
In this approach the aggregated logs are stored in an external Elasticsearch. This is the recommended approach in production as we don't have any overhead on the Internal Data Store when comparing to the API Gateway + Internal Data Store approach. Customers should create their own dashboard to view the logs. They should have their implementation to archive and purge the aggregated logs.
API Gateway + Internal Data Store + Terracotta Server
If the customer wishes to aggregate Terracotta server logs they can follow the below steps
There are 2 variants for this scenario.
Terracotta server is installed in the same machine where API Gateway is installed
- The terracotta logs by default will be available "(user.home)/terracotta/server-logs/." This folder can be changed by having a custom log data location in the tc-config.xml. Please refer Terracotta documentation for more information
- Disable log aggregation in "Administration → General→ Application logs" section
- Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- C:/Users/alice/terracotta/server-logs/terracotta-server.log
fields:
node: ${NODE}
fileType: TerracottaServerLogs
timezone: ${TIMEZONE}
fields_under_root: true
multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
multiline.negate: true
multiline.match: after
- Enable log aggregation in "Administration → General→ Application logs" section
- Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated
Terracotta server is installed in a different machine
- Change the tc-config.xml in Terracotta server to store the logs in a network location and start the Terracotta server
<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
<servers>
<server host="localhost" name="%h">
<data>C:/install/TerraCottaServer/Terracotta/server/bin/data</data>
<logs>//worstation01/share/tcserverlog</logs>
<offheap>
<enabled>true</enabled>
<maxDataSize>512m</maxDataSize>
</offheap>
</server>
<restartable enabled="true"/>
</servers>
</tc:tc-config>
- Disable log aggregation in "Administration → General→ Application logs" section
- Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- //mckmut02/share/tc-server-logs/terracotta-server.log
fields:
node: ${NODE}
fileType: TerracottaServerLogs
timezone: ${TIMEZONE}
fields_under_root: true
multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
multiline.negate: true
multiline.match: after
- Enable log aggregation in "Administration → General→ Application logs" section
- Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated
Configuring Log for Elasticsearch Client in API Gateway
API Gateway uses Elasticsearch REST client to connect to Internal Data Store or External Elasticsearch. By Default, the logs that are emitted by these REST clients are omitted to avoid over logging. To trouble shoot or diagnose elasticsearch calls from API Gateway, we can manually configure the log configuration for Elasticsearch REST client by following the below steps.
If you are using versions 10.3 or below
-
Open the file log4j.properties available at <install location>\IntegrationServer\instances\<instance name> directory.
-
Add the below lines at the end of the file.
-
log4j.logger.org.elasticsearch.client=DEBUG,ESRestClientLogger
-
log4j.appender.ESRestClientLogger=org.apache.log4j.RollingFileAppender
-
log4j.appender.ESRestClientLogger.File=logs/ESRestClientLogger.log
-
log4j.appender.ESRestClientLogger.MaxFileSize=50MB
-
log4j.appender.ESRestClientLogger.MaxBackupIndex=25
-
log4j.appender.ESRestClientLogger.layout=org.apache.log4j.PatternLayout
-
log4j.appender.ESRestClientLogger.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
-
-
After adding the lines, restart API Gateway.
-
The logs will be stored at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClientLogger.log
If you are using versions 10.5 or above
-
Open the log4j2.properties available <install location>\profiles/IS_default/configuration/logging/log4j2.properties
-
Add the below lines at the end of the file.
-
logger.
10
.name=org.elasticsearch.client
logger.
10
.additivity=
false
logger.
10
.level=info
logger.
10
.appenderRef.lar.ref=ESRestClient
logger.
11
.name=org.apache.http
logger.
11
.additivity=
false
logger.
11
.level=info
logger.
11
.appenderRef.lar.ref=ESRestClient
logger.
12
.name=org.apache.http.wire
logger.
12
.additivity=
false
logger.
12
.level=info
logger.
12
.appenderRef.lar.ref=ESRestClient
logger.
13
.name=org.apache.http.impl.conn
logger.
13
.additivity=
false
logger.
13
.level=info
logger.
13
.appenderRef.lar.ref=ESRestClient
appender.esrestclient.name=ESRestClient
appender.esrestclient.type=RollingFile
appender.esrestclient.fileName=<INSTALL_LOCATION>/IntegrationServer/instances/
default
/logs/ESRestClient.log
appender.esrestclient.filePattern=<INSTALL_LOCATION>/IntegrationServer/instances/
default
/logsESRestClient.log.%i
appender.esrestclient.layout.type=PatternLayout
appender.esrestclient.layout.pattern=%d [%t] %-5p %c %x - %m%n
appender.esrestclient.policies.type=Policies
appender.esrestclient.policies.size.type=SizeBasedTriggeringPolicy
appender.esrestclient.policies.size.size=10MB
appender.esrestclient.strategy.type=DefaultRolloverStrategy
appender.esrestclient.strategy.max=
10
#This is a custom Platform provided filter which matches all log messages that contain OSGi data in MDC (bundle.id, component.name, etc.)
appender.esrestclient.filter.osgi.type=LogServiceFilter
appender.esrestclient.filter.osgi.onMatch=DENY
appender.esrestclient.filter.osgi.onMismatch=NEUTRAL
-
-
After adding the lines restart API Gateway.
-
The logs will be available at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClient.log
In the above mentioned log, you can see the request sent to particular elasticsearch node and what is the response status code.