Handling of clear text passwords in external components of API Gateway

Knowledge base
, ,
1

Author: Asokan, Dhivakar <daso@softwareag.com>
Supported Versions: Since this deals only with the external components API Gateway version has no dependency here. The detailed support matrix of the component are available below

webMethods API Gateway tutorial

Overview of the tutorial

API Gateway uses multiple external components for its various functionalities like persistence, dashboards,log aggregation etc .Some of the external components have config files containing the product related configurations which will be picked during startup. Passwords are also part of this configurations files and user should configure the secret in these configuration files as plain text. Any user who have the access to the file system can view the passwords and access these components and can tamper the data. Since they are external components, we don't have control over their configuration files and startup procedure to mask these secrets and hence we planned to utilize their obfuscation method/settings storage for hiding the passwords from the yaml files.

Required knowledge

This tutorial assumes that the reader has:

  • Basic understanding about the working of API Gateway
  • Good understanding about the role of the external components in API Gateway
  • Basic configuration and administration of the external components

Why?

As per the security standards, any product should not have the passwords as plain text either in their code or in their configuration files. These secrets must have been stored in their secret store or at least these secret strings must be obfuscated and stored. This tutorial aims at providing the step by step process for saving the passwords of the below listed external components in to their keystore storage and referring them in the configuration files.

  • Elasticsearch
  • Kibana
  • Filebeat

Prerequisite steps

  • Installation of API Gateway (for on-premise deployment only)

Details

Keystore tool

Some settings are sensitive, and relying on file system permissions to protect their values is not sufficient. For this use case, Elastic stack provides a keystore tool, which may be password protected, and the keystore created stores all  the sensitive information. The password can be stored as a setting inside the keystore and the same can be referred in the yml file. During startup, the server will pick the value corresponding to the key configured and replace it. The detailed steps for creating a variable is explained in sections below. Below is the sample for using a setting available in the keystore.

Variable usage in yml

elasticsearch.username: ${setting_name}
eg: elasticsearch.password: ${sample_setting}

Points to remember

Few points to remember before trying out the keystore tool for Elasticsearch:

  • All commands here should be run as the user which will run Elasticsearch
  • Only some settings are designed to be read from the keystore
  • All the modifications to the keystore takes effect only after restarting the respective component
  • The keystore tool currently only provides obfuscation. In the future, password protection will be added
  • On a cluster setup, the settings need to be specified on each node. Currently, all the secure settings are node-specific settings that must have the same value on every node
  • All the settings created must be configured in the yaml file. Any unused setting will block the startup

Obfuscating passwords in Elasticsearch configuration file

Go to Elasticsearch installation directory.

Creating a keystore in Elasticsearch

The below command creates a keystore. All the secrets will be stored inside this store.

$> bin/elasticsearch-keystore create

The file elasticsearch.keystore will be created alongside elasticsearch.yml. If the same command is run again to create a another keystore, you will get a prompt to overwrite the existing keystore. 

Adding a setting in Elasticsearch

The below command adds a new setting to the existing keystore. Any unused setting in the keystore will block the startup process. Make sure you use all the setting in the yaml file or remove it before startup.

$> bin/elasticsearch-keystore add <setting_name>

For eg bin/elasticsearch-keystore add sample_setting

Setting- case sensitive: Till Elasticsearch version 6.2, you cannot have upper case in the setting name. The tool will not show any warning but when you list, you will get the warning as "Exception in thread "main" java.lang.SecurityException: Keystore has been corrupted or tampered with"

Listing all the settings in Elasticsearch

The below command lists all the settings (only keys)  from the existing keystore.

$> bin/elasticsearch-keystore list

Removing a setting  in Elasticsearch

The below command removes a setting from the existing keystore. 

$> bin/elasticsearch-keystore remove <setting_name>

For eg: bin/elasticsearch-keystore remove sample_setting

Obfuscating passwords in Kibana configuration file

Go to <SAG_Installation_Directory>/profiles/IS_<Instance_Name>/apigateway/dashboard/

Creating a keystore in Kibana

The below command creates a keystore. All the secrets will be stored inside this store.

$> bin/kibana-keystore create

The file kibana.keystore will be created inside the data folder. If the same command is run again to create a another keystore, you will get a prompt to overwrite the existing keystore. 

Adding a setting in Kibana

The below command adds a new setting to the existing keystore. Any unused setting in the keystore will block the startup process. Make sure you use all the setting in the yaml file or remove it before startup

$> bin/kibana-keystore add <setting_name>

For eg: bin/kibana-keystore add sample_setting

Listing all the settings in Kibana

The below command lists all the settings (only keys)  from the existing keystore.

$> bin/kibana-keystore list

Removing a setting  in Kibana

The below commands removes a setting from the existing keystore. 

$> bin/kibana-keystore remove <setting_name>

For eg: bin/kibana-keystore remove sample_setting

Obfuscating passwords in Filebeat configuration file

Go to <SAG_Installation_Directory>/profiles/IS_<Instance_Name>/apigateway/filebeat

Creating a keystore in Filebeat

The below command creates a keystore. All the secrets will be stored inside this store.

$> filebeat_apigateway keystore create

Filebeat creates the keystore in the directory defined by the path.config configuration setting.. If the same command is run again to create a another keystore, you will get a prompt to overwrite the existing keystore. 

Adding a setting in Filebeat

The below command adds a new setting to the existing keystore. Any unused setting in the keystore will block the startup process. Make sure, you use all the setting in the yaml file or remove it before startup

$> filebeat keystore add <setting_name>

For eg: bin/filebeat keystore add sample_setting

Listing all the settings in Filebeat

The below commands lists all the settings (only keys)  from the existing keystore.

$>  filebeat_apigateway filebeat keystore list

Removing a setting  in Filebeat

The below commands removes a setting from the existing keystore. 

$> filebeat_apigateway filebeat keystore remove <setting_name>

For eg: bin/filebeat keystore remove sample_setting

Workaround for non supported versions

For component versions which do not have keystore tool support, we can use the workaround of defining the settings in the environment variables and referring them in the yaml files. The environment variables can be created through a script just before executing the startup or we can define one at the system level and use it. Though the aim is to mask the plain text password, in this way, it will be available either in the script file or in the environment variable definition which we cannot avoid. This is just a workaround to avoid plain text passwords in the yml files. Kibana supports this way only from 6.3 where as other two support this from 5.x versions.

Creating a environment variable

set elasticsearch.username = elastic
set elasticsearch.password = changeme

Referring the environment variable in YAML

elasticsearch.username: ${elasticsearch.username}
elasticsearch.password: ${elasticsearch.password}

Supported versions

Products
Keystore supported version
Environment variable as workaround
Reference
Elasticsearch 5.3 and above (thumbs up) https://www.elastic.co/guide/en/elasticsearch/reference/5.3/secure-settings.html
Kibana 6.1 and above Above 6.3 version

https://www.elastic.co/guide/en/kibana/6.1/secure-settings.html

https://github.com/elastic/kibana/pull/16988
Filebeat 6.2 and above (thumbs up) https://www.elastic.co/guide/en/beats/filebeat/6.2/keystore.html

Component compatibility matrix

API Gateway
10.1
10.2
10.3
10.4
Elasticsearch 2.3.2 5.6.4 5.6.4 5.6.4
Kibana 4.5.1 5.6.4 5.6.4 5.6.9
Filebeat NA NA 6.0.1 6.0.1

Troubleshooting

Problem
Description
Resolution
Remarks

unknown secure setting while startup
uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.IllegalArgumentException:
unknown secure setting [tes] please
check that any required plugins are installed, or
check the breaking changes documentation for removed settings
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:136)
...
Caused by: java.lang.IllegalArgumentException: unknown secure setting
[tes] please check that any required plugins
are installed, or check the breaking changes documentation for removed settings
Remove the unused settings from the keystore for a healthy startup  

bin/elasticsearch-keystore list fails

 Exception in thread "main" java.lang.SecurityException: Keystore has been corrupted or tampered with Make sure that your setting does not have a upper case character. If yes, remove the setting and try again After ES 6.2, this is validated properly. The tool will not allow to insert these keys

References

Description
URL
Supported version

Installig Xpack

 https://www.elastic.co/guide/en/x-pack/5.6/installing-xpack.html > 5.6
Secure string in Kibana https://www.elastic.co/guide/en/kibana/current/secure-settings.html  
FileBeat Configuration https://www.elastic.co/guide/en/beats/filebeat/6.0/filebeat-configuration.html > 6.0
Securing FileBeat https://discuss.elastic.co/t/filebeat-best-way-to-secure-credentials/61704/17  
Securing FileBeat https://www.elastic.co/guide/en/beats/filebeat/current/keystore.html > 7.0
Logstash keystore https://www.elastic.co/guide/en/logstash/6.3/keystore.html > 6.3
X-pack settings support https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html > 6.2
Securing Elasticsearch with nginx https://www.minvolai.com/setting-up-a-secure-single-node-elasticsearch-server-behind-nginx/ > 2.x

Learn more

image.png