Starting API Gateway using externalized configurations

Knowledge base
, ,
1

Author: James, Clench Paign (clep@softwareag.com)
Supported Versions: 10.5 (on-premise deployment)

Overview of the tutorial

In this tutorial we will discuss about the details of externalized API Gateway configurations and how API Gateway can be started using those configurations. Let us go through the following points in detail.

  • Externalized configurations and the YAML templates
  • Master configuration YAML file and its usage
  • Properties file support for externalized configurations
  • Configuring multiple files and its effect
  • Default scenario for externalized configurations
Note: Currently there are two types of configurations in API Gateway, inter-component & cluster configurations and Product configurations. This feature covers the inter-component & cluster configurations. Inter-component configurations cover the communication between API Gateway, Elasticsearch, Kibana and Filebeat. Cluster configurations cover API Gateway, Elasticsearch and Terracotta cluster configurations.

Required knowledge

This tutorial assumes that the reader has,

  • a basic understanding of API Gateway as a product
  • a basic understanding of API Gateway and its communication with Elasticsearch, Kibana and Filebeat
  • a basic understanding of inter-component communications between Elasticsearch, Kibana and Filebeat
  • a basic understanding of Terracotta cluster and API Gateway server's communication with Terracotta Big memory
  • a basic understanding of Elasticsearch cluster

Why?

Currently the inter-component and cluster configurations of API Gateway are stored in different places within the product causing maintainability and operational overhead. Hence with this feature, the aforementioned configurations can be managed, provisioned from a central location supplied through one or more configuration files (YAML or Properties). This will help the API Gateway Administrator and operations teams to dynamically provision the API Gateway configurations enabling centralized configuration management, provisioning and scaling use cases.

Prerequisite steps

Complete the below prerequisite steps before starting API Gateway using externalized configurations.

  • Install API Gateway advanced edition of version 10.5 and above

Details

The API Gateway Administrator can provide the configurations in one or more YAML or properties files. He would then create a master configuration file in which the configuration files sources would be defined in order and place the file in a system defined location. The API Gateway, on startup, will first read from that master configuration file and start loading configurations from the defined configuration files. Using the configuration loaded from the configuration files, the values in the respective API Gateway internal configuration files (like uiconfiguration.properties, server.cnf, etc) will be updated. After the values from the configuration files are updated in API Gateway configuration space, the rest of the API Gateway package loading will run with the updated configuration values. The below sections explain how the API Gateway configurations can be externalized in files and using which how the API Gateway server can be started without modifying the respective internal configuration files.

Externalized configurations and their YAML templates

Using this feature the inter-component and cluster configurations can be externalized. The below sample file contains the configuration sections that an Administrator can use to externalize. The Administrator can provide only the necessary sections and settings in the configuration file which he would like to externalize. The given configurations will overwrite the existing configurations in the internal files only for the given configuration values. The configurations which the user doesn't provide will not be either overwritten or removed or commented out in and from the internal configuration files, instead, whatever the values the internal files have will become effective.

config.yml
apigw:
  elasticsearch:
    ...
  kibana:
    ...
  filebeat:
    ...
  cluster:
    ...

Elasticsearch configuration and details

Note: Install and run Elasticsearch of version 7.2 if an external instance needs to be configured for API Gateway

The Elasticsearch configurations contain all the necessary properties needed for an Elasticsearch HTTP client using which API Gateway connects to either an externally running Elasticsearch server or to the Internal data store which is run by API Gateway. The key configurations are given below.

tenantId - The API Gateway tenant id, using which the Elasticsearch indices are created for that tenant
hosts - A comma separated list of external Elasticsearch instances (for e.g "host1:9200,host2:9240")
autostart - This is an optional property. If not given, the value would be "false" so that API Gateway would connect to the given external Elasticsearch hosts rather starting the Internal data store. Suppose if the user wants to connect to the internal data store, he should set autostart to be "true" so that the API Gateway would start the internal data store. Note that the hosts should point to internal data store host and port in such case. The default hosts is "localhost:9240"
http - The basic authentication credentials and few HTTP connection specific properties
https - If "enabled" is set to true, API Gateway would use the given SSL related properties to connect to configured hosts
sniff - These properties helps in adding newly added Elasticsearch node in Elasticsearch cluster with providing its detail here. It will automatically manage the node details by contacting the Elasticsearch server
outboundproxy - Outbound proxy settings between API Gateway and Elasticsearch
clientHttpResponseSize - Maximum Response payload size in MB

A sample Elasticsearch configuration is given below:

elasticsearch.yml
apigw:
  elasticsearch:
    tenantId: apigateway
    hosts: localhost:9200
    autostart: false
    http:
      username: elastic
      password: changeme
      keepAlive: true
      keepAliveMaxConnections: 10
      keepAliveMaxConnectionsPerRoute: 100
      connectionTimeout: 1000
      socketTimeout: 10000
      maxRetryTimeout: 100000
    https:
      enabled: false
      keystoreFilepath: C:/softwares/elasticsearch/config/keystore-new.jks
      truststoreFilepath: C:/softwares/elasticsearch/config/truststore-new.ks
      keystoreAlias: root-ca
      keystorePassword: 6572b9b06156a0ff778c
      truststorePassword: manage
      enforceHostnameVerification: false
    sniff:
      enable: false
      timeInterval: 1000
    outboundProxy:
      enabled: false
      alias: somealias
    clientHttpResponseSize: 1001231

Kibana configuration and details

Note: Install Kibana of version 7.2 if an external instance needs to be configured for API Gateway

The Kibana configuration supports setting the Kibana server URL which can point to either the one which is run by API Gateway or any externally running server. It also contains the SSL certificate related settings that would be used to connect to the SSL protected Elasticsearch server. The key configurations are given below.

dashboardInstance - The Kibana server URL in the format <scheme>://<hostname>:<port> for e.g http://vmabc:5601
autostart - This is an optional property. If not given the value would be "false" so that API Gateway would connect to the given external Kibana server rather starting the Internal Kibana. Suppose if the user wants to connect to the internal Kibana, he should set autostart to be "true" so that the API Gateway would start the internal Kibana. Note that the hosts should point to internal Kibana server host and port in such case. The default value is "http://localhost:9405"
sslCA - A list of paths to the PEM file for the certificate authority for the Elasticsearch instance
sslCert - The path to the PEM format certificate for SSL client authentication
sslKey - The client certificate key used for client authentication. These files are used to verify the identity of Kibana to Elasticsearch server when it is SSL protected

A sample Kibana configuration is given below.

kibana.yml
apigw:
  kibana:
    dashboardInstance: http://localhost:9405
    autostart: true
    elasticsearch:
      sslCA: C:/softwares/elasticsearch/config/SAG-B1HPWT2.pem
      sslCert: C:/softwares/elasticsearch/config/SAG-B1HPWT2.crt
      sslKey: C:/softwares/elasticsearch/config/SAG-B1HPWT2.key

Filebeat configuration and details

The Filebeat configuration supports configuring the SSL certificate related settings that are used to connect to the SSL protected Elasticsearch server. The key configurations are given below.

sslCA - A list of paths to the PEM file for the certificate authority for the Elasticsearch instance
sslCert - The path to the PEM format certificate for SSL client authentication
sslKey - The client certificate key used for client authentication. These files are used to verify the identity of Kibana to Elasticsearch server when it is SSL protected

A sample Filebeat configuration is given below.

filebeat.yml
apigw:
  filebeat:
    output:
      elasticsearch:
        sslCA: C:/softwares/elasticsearch-5.6.4/config/SAG-B1HPWT2.pem
        sslCert: C:/softwares/elasticsearch-5.6.4/config/SAG-B1HPWT2.crt
        sslKey: C:/softwares/elasticsearch-5.6.4/config/SAG-B1HPWT2.key

Cluster configuration and details

Note: Install and run Terracotta server (a version which is compatible with API Gateway 10.5) for clustering API Gateway instances

The cluster configuration contains the Terracotta clustering as well as Elasticsearch clustering settings. The key configurations are given below.

aware, name, tsaUrls, sessTimeout, actionOnStartupError - All are Terracotta cluster settings that are set in the server watt properties
terracottaLicenseFileName - The Terracotta server license file name. The file should be present under <SAG>/common/conf. API Gateway would use this file for joining the Terracotta cluster
discoverySeedHosts - The discovery.seed_hosts property in the elasrticsearch.yml file of Internal data store
initialMasterNodes - The cluster.initial_master_nodes property in the elasrticsearch.yml file of Internal data store

A sample Cluster configuration is given below.

cluster.yml
apigw:
  cluster:
    aware: true
    name: APIGatewayTSAcluster
    tsaUrls: daeirnd33974:9510
    terracottaLicenseFileName: terracotta-license.key
    sessTimeout: 20
    actionOnStartupError: standalone
    esClusterName: SAG_EventDataStore
    discoverySeedHosts: daeirnd33974:9340,daeirnd33562:9300,daeirnd33974:8300
    initialMasterNodes : daeirnd33974_master

For terracottaLicenseFileName property a valid license file should be present in the <SAG>/common/conf location, otherwise the property wouldn't be effected.

Note: When cluster settings are given in the configuration files, the API Gateway server on startup would update the internal settings with the values from the configuration files but the node won't join the cluster. A server restart in required for the cluster settings to become effective and to join the cluster.

Master configuration YAML file and its usage

The master configuration YAML file contains the definitions of all the configuration files (both YAML and Properties types) and should be present in the system defined location SAG\IntegrationServer\instances\<instance>\packages\WmAPIGateway\resources\configuration. The file name is config-sources.yml which is also system defined. Any other file with different name will not be understood by the product. This is the initial file that would be picked up by API Gateway on startup and from there the API Gateway will learn the defined configuration files and their available locations and reads the configuration values from them. If any of the entry has an invalid file name or path that will be just ignored but the error would be logged into the API Gateway logs.

A sample master configuration file is given below.

config-sources.yml
sources:
- type: YAML
  allowEdit: true
  properties:
    location: elasticsearch.yml
- type: PROPERTIES
  allowEdit: false
  properties:
    location: C:/config/resources/kibana.properties

The properties of a configuration file source entry are explained below.

Property
Detail
type

This is the type of the configuration source. Applicable types would be YAML, PROPERTIES and CC_YAML.

YAML - a YAML configuration file

PROPERTIES - a properties configuration file

CC_YAML - a YAML configuration file which is reserved for Command Central updates
allowEdit

Enables whether this file can be updated in case of any edit from API Gateway. For now this helps to update hiding the passwords.

Valid values are true or false
properties

A miscellaneous set of properties which API Gateway would use to connect to the defined configuration source. For 10.5 release only configuration files are supported and hence a property location needs to be specified which points to the file path. It can be either an absolute path or a relative path. In case of relative path, the file would be located relative to the system defined location SAG\IntegrationServer\instances\<instance>\packages\WmAPIGateway\resources\configuration.

Note: For CC_YAML type the location is cc-config.yml which is a reserved one. This file would be the source of truth for all CRUD operations done from Command Central via REST APIs. It is strongly advised that this file should not be modified manually since it is supposed to be updated and read by and from CC by a CC Administrator.

Hiding clear text passwords in configuration files

In order to restrict any unknown user to have access to the passwords and credentials in the configuration files for any potential threats, the Administrator can enable hiding of such secrets by setting "allowEdit" flag to true in the master configuration file. The secret texts in the configuration files  would be stored in the Password manager and the clear texts in the files would be replaced with the Password manager keys after API Gateway is started and running. After this a user can see only the password keys in the files. On startup, API Gateway would retrieve the passwords for those settings from the Password manager using those keys and hence it is advised to not alter any of the password manager key values in the file. The passwords can be modified at any time and same will be replaced with the Password manager keys in the next API Gateway startup.

The below table contains the settings and their respective Password manager keys.

Setting
Password manager key replacement
 
apigw:
 elasticsearch:
   http:
     username: elastic
 @secure.elasticsearch.http.password 
apigw:
 elasticsearch:
   https:
     keystorePassword: 6572b9b06156a0ff778c
 @secure.elasticsearch.http.keystore.password 
apigw:
 elasticsearch:
   https:
     truststorePassword: 6572b9b06156a0ff778c
 @secure.elasticsearch.http.truststore.password

Properties file support for externalized configurations

The settings can be configured in Properties files as well. The property names would be in par with the YAML file configuration naming convention. For e.g the "tenantId" property in the YAML structure under apigw→elasticsearch can be specified as apigw.elasticsearch.tenantId in the properties file. Actually the property names in the YAML file configurations are delimited by a "." in the property file for forming the property name.

A sample Properties file is given below.

config.properties
apigw.elasticsearch.tenantId=default
apigw.elasticsearch.autostart=false
apigw.elasticsearch.hosts=vmabc\:9240
apigw.elasticsearch.http.password=admin123
apigw.elasticsearch.http.username=admin
apigw.kibana.dashboardInstance=http://localhost:9405
apigw.kibana.elasticsearch.sslCert=/path/to/your/client.crt

Configuring multiple files and its effect

The master configuration file can have many entries (0 to N) for defining multiple configuration files as configuration sources. When such file is used to start the API Gateway, the configuration values from all the files would be merged into a single effective configuration. In case of conflicts, i.e, if the same configuration value is present in two files then the value in the file which has higher preference would take priority. The order of preference is in the reverse order in which they are defined in the master configuration file, i.e, the configuration values that are defined in the last configuration file entry would have the highest preference. A sample use case is explained below.

For e.g. file1.yml has the below configurations.

file1.yml
apigw:
  elasticsearch:
    tenantId: default

file2.properties has the below property.

file2.properties
apigw.elasticsearch.tenantId=apigateway

file3.yml has the below configurations.

file3.yml
apigw:
  elasticsearch:
    http:
      username: admin
      password: admin123
  kibana:
    dashboardInstance: http://localhost:5601

Then the combined configuration that becomes effective would be as below.

Effective configuration - config.yml
apigw:
  elasticsearch:
    tenantId: apigateway
    http:
      username: admin
      password: admin123
  kibana:
    dashboardInstance: http://localhost:5601

Default scenario

By default when API Gateway is starting it would read the master configuration file to learn the defined configuration source files. If the file doesn't exist in the name config-sources.yml under the system defined location or the file is invalid, the API Gateway would fallback back to its default behavior and the values defined in the internal configuration files would become effective. Similarly if any of the configuration files doesn't exist or invalid, then those files will be simply ignored and API Gateway would start with the valid configuration files. The API Gateway server startup wouldn't be halted in any of the above scenarios, instead the error logs would be logged into API Gateway application logs for debugging purpose. Enable debug level of  the "Externalized Configuration" facility in the logging setting to see the logs.

The below are the sample logs for an API Gateway instance which is started using the externalized configurations.

server.log
[302]2019-08-16 11:19:02 IST [YAI.0013.8889I] [default][SAG-G43RXF2] Configuration loaded from configuration sources. APIGatewayConfig: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='apigw', hosts='localhost:9200', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[301]2019-08-16 11:19:02 IST [YAI.0013.8889I] [default][SAG-G43RXF2] APIGatewayConfig loaded from ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='apigw', hosts='null', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[300]2019-08-16 11:19:02 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Retrieving configuration from Properties file source: ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}
 
[299]2019-08-16 11:19:02 IST [YAI.0013.8889I] [default][SAG-G43RXF2] APIGatewayConfig loaded from ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='null', hosts='localhost:9200', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[298]2019-08-16 11:19:02 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Retrieving configuration from YAML file source: ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}
 
[297]2019-08-16 11:19:02 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Loading configuration from sources: [ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}, ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}]
 
[293]2019-08-16 11:19:01 IST [YAI.0013.8889I] [default][SAG-G43RXF2] Configuration loaded from configuration sources. APIGatewayConfig: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='apigw', hosts='localhost:9200', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[292]2019-08-16 11:19:01 IST [YAI.0013.8889I] [default][SAG-G43RXF2] APIGatewayConfig loaded from ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='apigw', hosts='null', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[291]2019-08-16 11:19:01 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Retrieving configuration from Properties file source: ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}
 
[290]2019-08-16 11:19:01 IST [YAI.0013.8889I] [default][SAG-G43RXF2] APIGatewayConfig loaded from ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}: APIGatewayConfig{elasticsearch=Elasticsearch{tenantId='null', hosts='localhost:9200', autostart='null', http=null, https=null, sniff=null, outboundProxy=null, clientHttpResponseSize=null, pendingRestart='null'}, kibana=null, filebeat=null, cluster=null}
 
[289]2019-08-16 11:19:01 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Retrieving configuration from YAML file source: ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}
 
[288]2019-08-16 11:19:01 IST [YAI.0013.8887D] [default][SAG-G43RXF2] Debug: Loading configuration from sources: [ConfigurationSource{type=YAML, allowEdit=true, properties={location=components.yml}}, ConfigurationSource{type=PROPERTIES, allowEdit=true, properties={location=components.properties}}]

system-settings.yml

The API Gateway ships with a default configuration file system-settings.yml which contains the default values of the inter-component and cluster configurations at the time of fresh installation. The API Administrator can anytime start the API Gateway with the original configuration values by referring this file in the master configuration file (config-sources.yml) with higher preference.

Limitations

  • The externalized configurations are now limited to inter-component and cluster configurations and other product level configurations are not supported now
  • When cluster settings are given in the configuration files, the API Gateway server on startup would update the internal settings with the values from the configuration files but the node won't join the cluster. A server restart in required for the cluster settings to become effective and to join the cluster

Troubleshooting

For any issues or misbehavior please check the below points.

  1. Check if the master config-sources.yml file exists and the file is a valid YAML file
  2. Check the configuration files locations are correctly configured in the master configuration file
  3. Check if the configuration files are valid YAML files
  4. Check the configuration files contain the right structure or naming for the settings outlined by the template
  5. Check whether the configured external instance (Elasticsearch or Kibana) is running before starting API Gateway
  6. Check the logs by enabling debug level of  the "Externalized Configuration" facility in the logging setting

References

Downloadable artifacts

Learn more

image.png

sample-config.properties (1.92 KB)

sample-config.yml (1.79 KB)