API Gateway logging - simplified

Clench_Paign_James3 · January 13, 2019, 4:46pm

webMethods API Gateway tutorial

Author: Chidambaram, Arun Dev (arunc@softwareag.com)
Supported Versions: 10.3 & above

Overview of the tutorial

API Gateway is dependent on various components like Integration Server,Internal Data Store, OSGI, Terracotta etc. Logs are generated by each of these components and stored in different locations. It becomes a cumbersome process to get these logs, set the log level and do the analysis when an error occurs in API Gateway. API Gateway in 10.3 simplifies this logging functionality so that customers can configure the log levels in a single place, download the logs and also enable log aggregation from different components. This tutorial will guide you on setting the log levels, downloading the logs, enabling log aggregation and also some of the best practices that should be followed.

Details

The log data is very important as it can be used in the following scenarios:

Monitoring the run time anomalies in API Gateway
Identifying the root cause of failures in API Gateway
Identifying the configuration issues in API Gateway
Detecting log patterns across the nodes in cluster

API Gateway is dependent on various components which has its own logging infrastructure.The various dependent components of API Gateway are as follows:

Integration Server
Internal Data Store
Platform (WSStack, OSGI etc)
Terracotta

API Gateway provides simplified solution with a one stop place to do the following actions:

To set the log level
Download the logs
Aggregate the logs from different components.
View the logs from different components in a single place and perform data mining

Platforms supported

This functionality is supported for both Windows and Unix platforms.

Setting the log levels for different components

The log levels for different components can be configured in the "Administration → General→ Application logs" section as shown below.

Explanation of log settings for the different components are shown below:

#	Log file type	Description	Available log levels	Source of the log file
#	Log file type	Description	Available log levels	Source of the log file
1	API Gateway	This setting changes the log level of API Gateway server. All the design time activities like creation of API,Application,Alias etc and the run-time activities like API invocation will be logged based on this setting	off, trace, debug, info, warn, error, fatal	<APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/server.log
2	Internal Data Store logs	This setting changes the log level of Internal Data Store. All API Gateway assets like API,Application,Alias,Policy etc are stored in this Internal Data Store	off, info, debug, warn, error, fatal	<APIGatewayInstallationDir>/InternalDataStore/logs/SAG_EventDataStore.log
3	Dashboard	This setting changes the log level of Dashboard. API Gateway internally uses Kibana to show graphs and visualize transnational data	silent, quiet, verbose	<APIGatewayInstallationDir>/profiles/IS_default/apigateway/dashboard/startup.log
4	Security	Enabling this logger will store the security events that occur in API Gateway	enable or disable	<APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSECURITY*.log
5	Session	Enabling this logger will store the session details like session id, session state, server id, timestamp etc for every login and logout happening in API Gateway	enable or disable	<APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSESSION*.log
6	Platform	This setting changes the log level of components that are available at the API Gateway platform like rampart,axis2,OSGI etc. Setting this log will require a API Gateway restart	off, trace, debug, info, warn, error, fatal	<APIGatewayInstallationDir>/GatewayTrunk/profiles/IS_default/logs/sag-osgi.log

Downloading diagnostics

The download diagnostic button is used to download diagnostics data for API Gateway. The diagnostics data that are collected are as follows.

Logs from different components like API Gateway Server, API Gateway UI, Internal Data Store, Security, Session, Platform , Wrapper.
Configurations and watt settings that are set in API Gateway
A thread dump of all the currently active threads.

All the above data are compressed and sent back. Since the data that is being collected is huge, it takes some time to download the diagnostic data.

Log aggregation

Enabling log aggregation will collect logs from the different components and store in Elasticsearch. There are 2 options to choose on where to store the logs.

API Gateway store

API Gateway comes with an Internal Data Store which is used to store assets in API Gateway.

External Elasticsearch

The external Elasticsearch server details such as host name, port, etc should be configured.

The Elasticsearch server configuration details are given below.

S.No	Fields	Description
1	Protocol	The type of protocol (HTTP, HTTPS) to use for the host and port combination.
2	Hostname	Specifies the host name or IP address of the machine on which external Elasticsearch resides.
3	Port	Specifies the port where external Elasticsearch server runs.
4	Indexname	Specifies the indexname which will be used by the external Elasticsearch to store the aggregated logs
5	Username	Specifies the Elasticsearch user ID for authenticating Elasticsearch when API Gateway communicates with it.
6	Password	Specifies the password of the Elasticsearch instance to be used for establishing communication between API Gateway and Elasticsearch.

Viewing the logs

Once log aggregation is enabled , API Gateway starts collecting the logs from various sources. These logs can be visualized and downloaded by navigating to "Analytics → Application logs". This analytics page will be rendered only if log aggregation destination is "API Gateway Store".Customer has to have their own tool to visualize the collected logs when "External Elasticsearch" destination is used to store aggregated logs.

Archiving, Purging and Restoring

Once the log aggregation is enabled , API Gateway starts collecting logs from various sources. These logs can be archived, purged and restored by going to "Administration → Manage data". The archive, purge and restore can be done from this screen only if the log aggregation destination is "API Gateway Store". Customer has to have their own logic for archiving, purging or restoring when "External Elasticsearch" destination is used to store aggregated logs.

Log configuration flavors

API Gateway internally uses Filebeat 6.0.1 to monitor the log files and forwards them to either API Gateway Store or external ElasticSearch. Filebeat is shipped along with API Gateway and itsconfiguration can be found in the following location "/<APIGateway Install Location>/profiles/IS_default/apigateway/filebeat/filebeat_apigateway.yml".

When log aggregation is enabled API Gateway starts Filebeat process in the background and it starts monitoring the log files and forwards them to either API Gateway Store or external Elasticsearch.

API Gateway + Internal Data Store

This is the Out of the box support available in API Gateway for log aggregation. Customers can just configure from the steps listed above and start the aggregating logs from different components.In this approach the same Internal Data Store is used for storing the API Gateway metadata like APIs, policies, configurations, run-time events etc. So if the log aggregation is also enabled then the Internal Data Store might be overloaded with too many requests and data might grow exponentially. So the administrators should keep track of the log aggregated data and periodically do purging.

API Gateway + External Elasticsearch

In this approach the aggregated logs are stored in an external Elasticsearch. This is the recommended approach in production as we don't have any overhead on the Internal Data Store when comparing to the API Gateway + Internal Data Store approach. Customers should create their own dashboard to view the logs. They should have their implementation to archive and purge the aggregated logs.

API Gateway + Internal Data Store + Terracotta Server

If the customer wishes to aggregate Terracotta server logs they can follow the below steps

There are 2 variants for this scenario.

Terracotta server is installed in the same machine where API Gateway is installed

The terracotta logs by default will be available "(user.home)/terracotta/server-logs/." This folder can be changed by having a custom log data location in the tc-config.xml. Please refer Terracotta documentation for more information
Disable log aggregation in "Administration → General→ Application logs" section
Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration

- type: log
  # Change to true to enable this prospector configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - C:/Users/alice/terracotta/server-logs/terracotta-server.log
  fields:
    node: ${NODE}
    fileType: TerracottaServerLogs
    timezone: ${TIMEZONE}
  fields_under_root: true
  multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
  multiline.negate: true
  multiline.match: after

Enable log aggregation in "Administration → General→ Application logs" section
Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated

Terracotta server is installed in a different machine

Change the tc-config.xml in Terracotta server to store the logs in a network location and start the Terracotta server

<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
 
  <servers>
    <server host="localhost" name="%h">
      <data>C:/install/TerraCottaServer/Terracotta/server/bin/data</data>
      <logs>//worstation01/share/tcserverlog</logs>   
      <offheap>
        <enabled>true</enabled>
        <maxDataSize>512m</maxDataSize>
      </offheap>
    </server>
    <restartable enabled="true"/>
  </servers>
</tc:tc-config>

Disable log aggregation in "Administration → General→ Application logs" section
Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration

- type: log
  # Change to true to enable this prospector configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - //mckmut02/share/tc-server-logs/terracotta-server.log
  fields:
    node: ${NODE}
    fileType: TerracottaServerLogs
    timezone: ${TIMEZONE}
  fields_under_root: true
  multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
  multiline.negate: true
  multiline.match: after

Enable log aggregation in "Administration → General→ Application logs" section
Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated

Configuring Log for Elasticsearch Client in API Gateway

API Gateway uses Elasticsearch REST client to connect to Internal Data Store or External Elasticsearch. By Default, the logs that are emitted by these REST clients are omitted to avoid over logging. To trouble shoot or diagnose elasticsearch calls from API Gateway, we can manually configure the log configuration for Elasticsearch REST client by following the below steps.

If you are using versions 10.3 or below

Open the file log4j.properties available at <install location>\IntegrationServer\instances\<instance name> directory.
Add the below lines at the end of the file.
- log4j.logger.org.elasticsearch.client=DEBUG,ESRestClientLogger
- log4j.appender.ESRestClientLogger=org.apache.log4j.RollingFileAppender
- log4j.appender.ESRestClientLogger.File=logs/ESRestClientLogger.log
- log4j.appender.ESRestClientLogger.MaxFileSize=50MB
- log4j.appender.ESRestClientLogger.MaxBackupIndex=25
- log4j.appender.ESRestClientLogger.layout=org.apache.log4j.PatternLayout
- log4j.appender.ESRestClientLogger.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n
After adding the lines, restart API Gateway.
The logs will be stored at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClientLogger.log

If you are using versions 10.5 or above

Open the log4j2.properties available <install location>\profiles/IS_default/configuration/logging/log4j2.properties
Add the below lines at the end of the file.
- logger.10.name=org.elasticsearch.client
  
  logger.10.additivity=false
  
  logger.10.level=info
  
  logger.10.appenderRef.lar.ref=ESRestClient
  
  logger.11.name=org.apache.http
  
  logger.11.additivity=false
  
  logger.11.level=info
  
  logger.11.appenderRef.lar.ref=ESRestClient
  
  logger.12.name=org.apache.http.wire
  
  logger.12.additivity=false
  
  logger.12.level=info
  
  logger.12.appenderRef.lar.ref=ESRestClient
  
  logger.13.name=org.apache.http.impl.conn
  
  logger.13.additivity=false
  
  logger.13.level=info
  
  logger.13.appenderRef.lar.ref=ESRestClient
  
  appender.esrestclient.name=ESRestClient
  
  appender.esrestclient.type=RollingFile
  
  appender.esrestclient.fileName=<INSTALL_LOCATION>/IntegrationServer/instances/default/logs/ESRestClient.log
  
  appender.esrestclient.filePattern=<INSTALL_LOCATION>/IntegrationServer/instances/default/logsESRestClient.log.%i
  
  appender.esrestclient.layout.type=PatternLayout
  
  appender.esrestclient.layout.pattern=%d [%t] %-5p %c %x - %m%n
  
  appender.esrestclient.policies.type=Policies
  
  appender.esrestclient.policies.size.type=SizeBasedTriggeringPolicy
  
  appender.esrestclient.policies.size.size=10MB
  
  appender.esrestclient.strategy.type=DefaultRolloverStrategy
  
  appender.esrestclient.strategy.max=10
  
  #This is a custom Platform provided filter which matches all log messages that contain OSGi data in MDC (bundle.id, component.name, etc.)
  
  appender.esrestclient.filter.osgi.type=LogServiceFilter
  
  appender.esrestclient.filter.osgi.onMatch=DENY
  
  appender.esrestclient.filter.osgi.onMismatch=NEUTRAL
After adding the lines restart API Gateway.
The logs will be available at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClient.log

In the above mentioned log, you can see the request sent to particular elasticsearch node and what is the response status code.