API Gateway logging - simplified

webMethods API Gateway tutorial

Author: Chidambaram, Arun Dev (arunc@softwareag.com)
Supported Versions: 10.3 & above

Overview of the tutorial

API Gateway is dependent on various components like Integration Server,Internal Data Store, OSGI, Terracotta etc. Logs are generated by each of these components and stored in different locations. It becomes a cumbersome process to get these logs, set the log level and do the analysis when an error occurs in API Gateway. API Gateway in 10.3 simplifies this logging functionality so that customers can configure the log levels in a single place, download the logs and also enable log aggregation from different components. This tutorial will guide you on setting the log levels, downloading the logs, enabling log aggregation and also some of the best practices that should be followed.

Details

The log data is very important as it can be used in the following scenarios:

  • Monitoring the run time anomalies in API Gateway
  • Identifying the root cause of failures in API Gateway
  • Identifying the configuration issues in API Gateway
  • Detecting log patterns across the nodes in cluster

API Gateway is dependent on various components which has its own logging infrastructure.The various dependent components of API Gateway are as follows:

  • Integration Server
  • Internal Data Store
  • Platform (WSStack, OSGI etc)
  • Terracotta

API Gateway provides simplified solution with a one stop place to do the following actions:

  • To set the log level
  • Download the logs
  • Aggregate the logs from different components.
  • View the logs from different components in a single place and perform data mining

Platforms supported

This functionality is supported for both Windows and Unix platforms.

Setting the log levels for different components

The log levels for different components can be configured in the "Administration → General→ Application logs" section as shown below.

Explanation of log settings for the different components are shown below:

#
Log file type
Description
Available log levels
Source of the log file
#
Log file type
Description
Available log levels
Source of the log file
1 API Gateway

This setting changes the log level of API Gateway server. All the design time activities like creation of API,Application,Alias etc and the run-time activities like API invocation will be logged based on this setting

off, trace, debug, info, warn, error, fatal <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/server.log
2 Internal Data Store logs

This setting changes the log level of Internal Data Store. All API Gateway assets like API,Application,Alias,Policy etc are stored in this Internal Data Store

off, info, debug, warn, error, fatal <APIGatewayInstallationDir>/InternalDataStore/logs/SAG_EventDataStore.log
3 Dashboard

This setting changes the log level of Dashboard. API Gateway internally uses Kibana to show graphs and visualize transnational data

silent, quiet, verbose <APIGatewayInstallationDir>/profiles/IS_default/apigateway/dashboard/startup.log
4 Security

Enabling this logger will store the security events that occur in API Gateway

enable or disable <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSECURITY*.log
5 Session

Enabling this logger will store the session details like session id, session state, server id, timestamp etc for every login and logout

happening in API Gateway

enable or disable <APIGatewayInstallationDir>/IntegrationServer/instances/default/logs/WMSESSION*.log
6 Platform

This setting changes the log level of components that are available at the API Gateway platform like rampart,axis2,OSGI etc. Setting this log will require a API Gateway restart

off, trace, debug, info, warn, error, fatal <APIGatewayInstallationDir>/GatewayTrunk/profiles/IS_default/logs/sag-osgi.log

Downloading diagnostics

The download diagnostic button is used to download diagnostics data for API Gateway. The diagnostics data that are collected are as follows.

  • Logs from different components like API Gateway Server, API Gateway UI, Internal Data Store, Security, Session, Platform , Wrapper.
  • Configurations and watt settings that are set in API Gateway
  • A thread dump of all the currently active threads.

All the above data are compressed and sent back. Since the data that is being collected is huge, it takes some time to download the diagnostic data.

Log aggregation

Enabling log aggregation will collect logs from the different components and store in Elasticsearch. There are 2 options to choose on where to store the logs.

API Gateway store

API Gateway comes with an Internal Data Store which is used to store assets in API Gateway.

External Elasticsearch

The external Elasticsearch server details such as host name, port, etc should be configured.

The Elasticsearch server configuration details are given below.

S.No
Fields
Description
1 Protocol The type of protocol (HTTP, HTTPS) to use for the host and port combination.
2 Hostname Specifies the host name or IP address of the machine on which external Elasticsearch resides.
3 Port Specifies the port where external Elasticsearch server runs.
4 Indexname Specifies the indexname which will be used by the external Elasticsearch to store the aggregated logs
5 Username Specifies the Elasticsearch user ID for authenticating Elasticsearch when API Gateway communicates with it.
6 Password Specifies the password of the Elasticsearch instance to be used for establishing communication between API Gateway and Elasticsearch.

Viewing the logs

Once log aggregation is enabled , API Gateway starts collecting the logs from various sources. These logs can be visualized and downloaded by navigating to "Analytics → Application logs". This analytics page will be rendered only if log aggregation destination is "API Gateway Store".Customer has to have their own tool to visualize the collected logs when "External Elasticsearch" destination is used to store aggregated logs.

Archiving, Purging and Restoring

Once the log aggregation is enabled , API Gateway starts collecting logs from various sources. These logs can be archived, purged and restored by going to "Administration → Manage data". The archive, purge and restore can be done from this screen only if the log aggregation destination is "API Gateway Store". Customer has to have their own logic for archiving, purging or restoring when "External Elasticsearch" destination is used to store aggregated logs.

Log configuration flavors

API Gateway internally uses Filebeat 6.0.1 to monitor the log files and forwards them to either API Gateway Store or external ElasticSearch. Filebeat is shipped along with API Gateway and itsconfiguration can be found in the following location "/<APIGateway Install Location>/profiles/IS_default/apigateway/filebeat/filebeat_apigateway.yml".

When log aggregation is enabled API Gateway starts Filebeat process in the background and it starts monitoring the log files and forwards them to either API Gateway Store or external Elasticsearch.

API Gateway + Internal Data Store

This is the Out of the box support available in API Gateway for log aggregation. Customers can just configure from the steps listed above and start the aggregating logs from different components.In this approach the same Internal Data Store is used for storing the API Gateway metadata like APIs, policies, configurations, run-time events etc. So if the log aggregation is also enabled then the Internal Data Store might be overloaded with too many requests and data might grow exponentially. So the administrators should keep track of the log aggregated data and periodically do purging.

API Gateway + External Elasticsearch

In this approach the aggregated logs are stored in an external Elasticsearch. This is the recommended approach in production as we don't have any overhead on the Internal Data Store when comparing to the API Gateway + Internal Data Store approach. Customers should create their own dashboard to view the logs. They should have their implementation to archive and purge the aggregated logs.

API Gateway + Internal Data Store + Terracotta Server

If the customer wishes to aggregate Terracotta server logs they can follow the below steps

There are 2 variants for this scenario.

Terracotta server is installed in the same machine where API Gateway is installed

  • The terracotta logs by default will be available "(user.home)/terracotta/server-logs/." This folder can be changed by having a custom log data location in the tc-config.xml. Please refer Terracotta documentation for more information
  • Disable log aggregation in "Administration → General→ Application logs" section
  • Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration
- type: log
  # Change to true to enable this prospector configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - C:/Users/alice/terracotta/server-logs/terracotta-server.log
  fields:
    node: ${NODE}
    fileType: TerracottaServerLogs
    timezone: ${TIMEZONE}
  fields_under_root: true
  multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
  multiline.negate: true
  multiline.match: after
  • Enable log aggregation in "Administration → General→ Application logs" section
  • Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated

Terracotta server is installed in a different machine

  • Change the tc-config.xml in Terracotta server to store the logs in a network location and start the Terracotta server
<?xml version="1.0" encoding="UTF-8"?>
<tc:tc-config xmlns:tc="http://www.terracotta.org/config">
 
  <servers>
    <server host="localhost" name="%h">
      <data>C:/install/TerraCottaServer/Terracotta/server/bin/data</data>
      <logs>//worstation01/share/tcserverlog</logs>   
      <offheap>
        <enabled>true</enabled>
        <maxDataSize>512m</maxDataSize>
      </offheap>
    </server>
    <restartable enabled="true"/>
  </servers>
</tc:tc-config>
  • Disable log aggregation in "Administration → General→ Application logs" section
  • Edit the "<APIGateway Install Dir>\profiles\IS_default\apigateway\filebeat\filebeat_template.yml" to add the following content below the DashboardLogs log confiuration
- type: log
  # Change to true to enable this prospector configuration.
  enabled: true
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
  - //mckmut02/share/tc-server-logs/terracotta-server.log
  fields:
    node: ${NODE}
    fileType: TerracottaServerLogs
    timezone: ${TIMEZONE}
  fields_under_root: true
  multiline.pattern: (([\s]+)20[0-9]{2}-)|20[0-9]{2}-
  multiline.negate: true
  multiline.match: after
  • Enable log aggregation in "Administration → General→ Application logs" section
  • Go the Analytics → Application logs and verify whether Terracotta server logs are aggregated

Configuring Log for Elasticsearch Client in API Gateway

API Gateway uses Elasticsearch REST client to connect to Internal Data Store or External Elasticsearch. By Default, the logs that are emitted by these REST clients are omitted to avoid over logging. To trouble shoot or diagnose elasticsearch calls from API Gateway, we can manually  configure the log configuration for Elasticsearch REST client by following the below steps.

If you are using versions 10.3 or below

  1. Open the file log4j.properties available at <install location>\IntegrationServer\instances\<instance name> directory.

  2. Add the below lines at the end of the file.

    • log4j.logger.org.elasticsearch.client=DEBUG,ESRestClientLogger

    • log4j.appender.ESRestClientLogger=org.apache.log4j.RollingFileAppender

    • log4j.appender.ESRestClientLogger.File=logs/ESRestClientLogger.log

    • log4j.appender.ESRestClientLogger.MaxFileSize=50MB

    • log4j.appender.ESRestClientLogger.MaxBackupIndex=25

    • log4j.appender.ESRestClientLogger.layout=org.apache.log4j.PatternLayout

    • log4j.appender.ESRestClientLogger.layout.ConversionPattern=%d [%t] %-5p %c %x - %m%n

  3. After adding the lines, restart API Gateway.

  4. The logs will be stored at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClientLogger.log

If you are using versions 10.5 or above

  1. Open the log4j2.properties available <install location>\profiles/IS_default/configuration/logging/log4j2.properties

  2. Add the below lines at the end of the file.

    • logger.10.name=org.elasticsearch.client
      logger.10.additivity=false
      logger.10.level=info
      logger.10.appenderRef.lar.ref=ESRestClient
       
      logger.11.name=org.apache.http
      logger.11.additivity=false
      logger.11.level=info
      logger.11.appenderRef.lar.ref=ESRestClient
       
      logger.12.name=org.apache.http.wire
      logger.12.additivity=false
      logger.12.level=info
      logger.12.appenderRef.lar.ref=ESRestClient
       
      logger.13.name=org.apache.http.impl.conn
      logger.13.additivity=false
      logger.13.level=info
      logger.13.appenderRef.lar.ref=ESRestClient
       
      appender.esrestclient.name=ESRestClient
      appender.esrestclient.type=RollingFile
      appender.esrestclient.fileName=<INSTALL_LOCATION>/IntegrationServer/instances/default/logs/ESRestClient.log
      appender.esrestclient.filePattern=<INSTALL_LOCATION>/IntegrationServer/instances/default/logsESRestClient.log.%i
      appender.esrestclient.layout.type=PatternLayout
      appender.esrestclient.layout.pattern=%d [%t] %-5p %c %x - %m%n
      appender.esrestclient.policies.type=Policies
      appender.esrestclient.policies.size.type=SizeBasedTriggeringPolicy
      appender.esrestclient.policies.size.size=10MB
      appender.esrestclient.strategy.type=DefaultRolloverStrategy
      appender.esrestclient.strategy.max=10
      #This is a custom Platform provided filter which matches all log messages that contain OSGi data in MDC (bundle.id, component.name, etc.)
      appender.esrestclient.filter.osgi.type=LogServiceFilter
      appender.esrestclient.filter.osgi.onMatch=DENY
      appender.esrestclient.filter.osgi.onMismatch=NEUTRAL
  3. After adding the lines restart API Gateway.

  4. The logs will be available at <install folder>\IntegrationServer\instances\<instance name>\logs\ESRestClient.log

In the above mentioned log, you can see the request sent to particular elasticsearch node and what is the response status code. 

image.png