Securing Elasticsearch for API Gateway 10.5

webMethods API Gateway tutorial

Author: Rizwan, Mohammed (mriz@softwareag.com)
Supported Versions: 10.5

Overview of the tutorial

This tutorial helps to understand how the InternalDataStore (or a simple Elasticsearch instance) can be secured using Search Guard, an Elasticsearch plugin that offers encryption, authentication and authorization. In this context, this plugin helps to secure the InternalDataStore by exposing it over HTTPS protocol and enabling a basic authentication by configuring users.

This tutorial covers the below steps in detail.

  • Installation and initialization of Search Guard plugin
  • Adding users for basic authentication
  • Changing Kibana configurations to securely connect to Elasticsearch
  • Configuring API Gateway to securely connect to Elasticsearch
  • Configuring Search Guard with user generated certificates

Required knowledge

The reader has to have:

  • a basic understanding of API Gateway and its communication with InternalDataStore for storing data
  • a basic understanding of Kibana and its communication with InternalDataStore for rendering the dashboards in API Gateway

Why?

InternalDataStore, a wrapper over Elasticsearch, being the back-end store for API Gateway, it is necessary to protect this store from attackers and other misuses which would eventually damage the entire functionality of the API Gateway in a way that it can not be recoverable. Hence it is almost important to protect the InternalDataStore and this security aspect can be achieved by using Search Guard, a third party plugin, which can be plugged into the Elasticsearch server for providing security functionalities.

Prerequisite steps

Complete the below prerequisites steps before get into the details of securing InternalDataStore of API Gateway using SearchGuard plugin.

  • Install API Gateway advanced edition of version 10.2 and above

Details

The below sections describe the details of securing InternalDataStore using Search Guard plugin by installing the plugin and adding the necessary configurations both in InternalDataStore, Kibana and API Gateway.

Step 1: Installation and initialization of Search Guard plugin

This section explains how to install the Search Guard plugin and make the InternalDataStore ready for enabling security.

Shutdown the API Gateway. Open <SAG_Root>/InternalDataStore /bin/enable_ssl.bat and comment out the last line ..\plugins\search-guard-5\tools\sgadmin.bat and save.

Copy sagconfig from <SAG_Root>/IntegrationServer/instances/<Instance_Name>/packages/WmAPIGateway/config/resources/elasticsearch to <SAG_Root>/InternalDataStore. Then copy the certificates node-0-keystore.jks and truststore.jks from <SAG_Root>/InternalDataStore/sagconfig to <SAG_Root>/InternalDataStore/config.

Open <SAG_Root>/InternalDataStore/config/elasticsearch.yml file. Remove all the properties that start with "searchguard" if such present and add the Search Guard properties as given below and save the file.

Search Guard properties

searchguard.ssl.transport.keystore_type: JKS
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_alias: cn=node-0
searchguard.ssl.transport.keystore_password: a362fbcce236eb098973
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_alias: root-ca-chain
searchguard.ssl.transport.truststore_password: 2c0820e69e7dd5356576
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enable_openssl_if_available: true

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_type: JKS
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.keystore_alias: cn=node-0
searchguard.ssl.http.keystore_password: a362fbcce236eb098973
searchguard.ssl.http.truststore_type: JKS
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_alias: root-ca-chain
searchguard.ssl.http.truststore_password: 2c0820e69e7dd5356576
searchguard.ssl.http.clientauth_mode: OPTIONAL

searchguard.enable_snapshot_restore_privilege: true
searchguard.check_snapshot_restore_write_privileges: true
searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

searchguard.authcz.admin_dn:
  - "CN=sgadmin"

Search Guard properties

A brief description of each Search Guard property is given below.

Item Desc Possible Values Default
TRANSPORT ( 2-Way authentication is enabled by default)
searchguard.ssl.transport.keystore_type Type of keystore JKS, PKCS12 JKS
searchguard.ssl.transport.keystore_filepath Location where the keystore    
searchguard.ssl.transport.keystore_alias Keystore entry name if there are more than one entries.    
searchguard.ssl.transport.keystore_password Password to access keystore.    
searchguard.ssl.transport.truststore_type Type of truststore JKS, PKCS12 JKS
searchguard.ssl.transport.truststore_filepath Location where the truststore    
searchguard.ssl.transport.truststore_alias Truststore entry name if there are more than one entries.    
searchguard.ssl.transport.truststore_password Password to access truststore.    
searchguard.ssl.transport.enforce_hostname_verification

If true, the hostname mentioned in certificate will be validated. We set this to false as ours is general purpose self signed certs.

true / false true
searchguard.ssl.transport.resolve_hostname

Applicable only if above property is true. If true, the hostname will be resolved against the DNS server. We set this to false as ours is general purpose self signed certs.

true / false true
searchguard.ssl.transport.enable_openssl_if_available Use if OpenSSL is available instead of JDK SSL. true / false true
HTTP
searchguard.ssl.http.enabled To enable the SSL for REST interface ( HTTP). We set this to true. true / false false
searchguard.ssl.http.keystore_type Type of keystore JKS, PKCS12 JKS
searchguard.ssl.http.keystore_filepath Location where the keystore    
searchguard.ssl.http.keystore_alias Keystore entry name if there are more than one entries.    
searchguard.ssl.http.keystore_password Password to access keystore.    
searchguard.ssl.http.truststore_type Type of truststore JKS, PKCS12 JKS
searchguard.ssl.http.truststore_filepath Location where the truststore    

searchguard.ssl.http.truststore_alias

Truststore entry name if there are more than one entries    
searchguard.ssl.http.truststore_password Password to access truststore.    
searchguard.ssl.http.clientauth_mode

Option to enable 2-way authentication.

REQUIRE : Client will be challenged for client certificate.

OPTIONAL : Will be used if client certificate is available

NONE : Ignore client certificate even if it's available.

REQUIRE,

OPTIONAL,

NONE

OPTIONAL
Search Guard Admin
searchguard.authcz.admin_dn

Search Guard maintains all the data in a index called "searchguard". This is accessible to only users ( client cert which will be passed in sdadmin command) configured here.

   
Misc

searchguard.cert.oid

 

All certificates used by the nodes on transport level need to have the oid field set to a specific value.This oid value is checked by

Search Guard to identify if an incoming request comes from a trusted node in the cluster or not. In the former case, all actions are allowed,

in the latter case, privilege checks apply. Plus, the oid is also checked whenever a node wants to join the cluster.

 

'1.2.3.4.5.5'

searchguard.config_index_name

Index where all the security configuration is stored. It's not confgurable for now but in the future  

searchguard

 

searchguard.enable_snapshot_restore_privilege
searchguard.check_snapshot_restore_write_privileges
To perform snapshot and restore operations, a user needs to have special privileges assigned. These two lines enable these privileges    
searchguard.restapi.roles_enabled Tells Search Guard which Search Guard roles can access the REST Management API to perform changes to the configuration    

Now run <SAG_Root>/InternalDataStore/bin/enable_ssl.bat file. This will Install Search Guard plugin and start the InternalDataStore.

Now shutdown and restart the InternalDataStore. Go to <SAG_Root>/InternalDataStore/plugins/search-guard-7/tools and execute the below command.

sgadmin.bat -cd ..\..\..\sagconfig\ -ks ..\..\..\sagconfig\sgadmin-keystore.jks -kspass 49fc2492ebbcfa7cfc5e -ts ..\..\..\sagconfig\truststore.jks -tspass 2c0820e69e7dd5356576 -nhnv -p 9340 -cn SAG_EventDataStore

This will Initialize the InternalDataStore. Refer the image given below.

Note: The Password -kspass 49fc2492ebbcfa7cfc5e and –tspass 2c0820e69e7dd5356576 are default passwords for keystore and truststore and the default certificates are not safe for production.

Step 2: Adding users for basic authentication

In this section let us see how an user can be provisioned in the Search Guard plugin to provide a http basic authentication.

Go to <SAG_Root>\InternalDataStore\sagconfig and open sg_roles_mapping.yml file. Add the username (e.g. TestUser) in the users list as given in the below image.

Before providing the password it must be converted into hash code. To generate the hash code for your password , execute <SAG_Root>\InternalDataStore\plugins\search-guard-7\tools\hash.bat. Type the password, press enter and it will generate the hash code as given in below image.

Now open sg_internal_users.yml file from <SAG_Root>\InternalDataStore\sagconfig  and add the username and password as given in the below image.

After completing the configurations, initialize Search Guard plugin again by executing the sgadmin.bat command as we did before. After initialization shutdown and restart the InternalDataStore. Now the InternalDataStore is running securely by opening a HTTPS port and requesting basic authentication details.

Step 3: Changing Kibana configurations to connect to Elasticsearch

Now we need to modify the Kibana server configurations to connect to the Elasticsearch securely over HTTPS port using basic authentication details.

  • Open kibana.yml file from the directory <SAG-Home>\profiles\IS_default\apigateway\dashboard\config
  • Remove the comments and update the following properties given below
    • elasticsearch.username: TestUser
    • elasticsearch.password: TestUser@123
    • elasticsearch.ssl.verificationMode: certificate
    • elasticsearch.ssl.certificateAuthorities: <file path of your root-ca.pem certificate>
    • elasticsearch.url: https://<hostname>:9240
  • After changing these configurations open uiconfiguration.properties file at <SAG_Home>\profiles\IS_default\apigateway\config and set apigw.kibana.autostart to false

Sample kibana.yml file image is given below.

Step 4: Changing API Gateway configurations to connect to Elasticsearch

Finally change the API Gateway configurations to connect to InternalDataStore securely. Follow the below steps.

  • Go to <SAG_HOME>/IntegrationServer/instances/default/packages/WmAPIGateway/config/resources/elasticsearch and open config.properties file. Uncomment the following properties and provide the values for them as below
    • pg.gateway.elasticsearch.http.username=TestUser
    • pg.gateway.elasticsearch.http.password=TestUser@123
    • pg.gateway.elasticsearch.https.truststore.filepath=C:/SoftwareAG-A-105/InternalDataStore/sagconfig/truststore.jks
    • pg.gateway.elasticsearch.https.truststore.password=2c0820e69e7dd5356576
    • pg.gateway.elasticsearch.https.enabled=true
  • After making all the configurations, start the InternalDataStore manually
  • When InternalDataStore is up and running, start the Kibana server manually (To start kibana server manually run kibana.bat file at <SAG_HOME>/profiles\IS_default\apigateway\dashboard\bin)
  • Now start the API Gateway and you would be able to login and create APIs. Analytics page should be accessible without any challenge window for user credentials

Step 5: Configuring Search Guard with user generated certificates (optional)

The API Provider can generate his own certificates to be used with Search Guard instead of the default certificates that are shipped with API Gateway. To achieve this, Search Guard provides an offline TLS tool, using which all the required certificates can be generated for running Search Guard in a production environment. Follow the below steps to generate certificates using Search Guard offline TLS tool.

Download the tool zip file from https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22search-guard-tlstool%22 (the same can be found in the attachments) and unzip it to a directory. Create a yaml file at <OfflineTool Installation Directory>\config. On executing the TLS tool command, it will read the node and certificate configuration settings from this yaml file and outputs the generated files in a configured directory. The sample file is given below. Please find the attachments for the sample file, Certificates.yml.

After configuring the yaml file run the below command to generate the required certificates.
<OfflineTool Installation Directory>/tools/sgtlstool.bat -c ../config/Demo.yml -ca -crt
The generated certificates can be outputted at <OfflineTool Installation Directory>/tools/out 

For further details on this refer the link provided in the References. It is mandatory to copy the certificates listed below from <OfflineTool Installation Directory>/tools/out to <SAG_Root>/InternalDataStore/config folder.

  • Dinesh-node-1.key
  • Dinesh-node-1.pem
  • Dinesh-node-1_http.pem
  • Dinesh-node-1_http.key
  • Dinesh-client.pem
  • Dinesh-client.key
  • root-ca.pem
  • root-ca.key

Now configure the generated certificates in the InternalDataStore elasticsearch.yml file.

After completing all the above steps start the InternalDataStore manually. After InternalDataStore is up, since the Search Guard is not initialized with the latest certificates, a log message would be shown saying that the Search Guard is not initialized. The following section provide the steps to initialize the Search Guard with the generated certificates.

  • Open command prompt and change directory to <SAG-Home>\InternalDataStore \plugins\search-guard-5\tools
  • Execute the command sgadmin.sh -cd ..\sagconfig –nhnv –icl –cacert ..\..\..\config\root-ca.pem -cert ..\..\..\config\Dinesh-client.pem -key ..\..\..\config\Dinesh-client.key -keypass <your certificate password> -p 9340 . A log "Done with success" will show up
  • Now shutdown and restart the InternalDataStore. The InternalDataStore will now use the generated certificates for SSL communication

Additional Steps while using user generated certificates

  1. Using a key tool import generated pem certificate "root-ca.pem" into truststore.
  2. Steps for generating a empty  truststore:
    • keytool -genkey -keyalg RSA -alias endeca -keystore truststore.jks
    • keytool -delete -alias endeca -keystore truststore.jks
  3. Command for importing pem file into truststore
    1. keytool -import -v -trustcacerts -alias endeca-ca -file root-ca.pem -keystore truststore.jks 

After which, use the generated truststore in step 3.

Troubleshooting

# Problem Solution
1

Search Guard may not be initialized. Initialize Search Guard Using sgadmin tool

Make sure that Search Guard is initialized properly using the steps given above
2

ES-Log contains entries like the following:

[2020-01-01T12:00:00,097] [INFO] [c.f.s.c.ConfigurationRepository] Search Guard License Type: TRIAL, valid

[2020-01-01T12:00:00,097] [WARN] [c.f.s.c.ConfigurationRepository] Your Search Guard license expires in 17 days.

The Community Edition does not require a license. To use it, just install the Enterprise Edition and disable all commercial features by adding the following line to elasticsearch.yml and restart the node:

searchguard.enterprise_modules_enabled: false

Setting this flag will disable any commercial module or custom authentication domain and will run only the free Community features of Search Guard.

3

Error in ElasticSearch:
Speaks http plaintext instead of SSL, will close the channel

It means Kibana or API Gateway is still communicating with Elasticsearch in HTTP.
Configure the https protocol and certificates in kibana.yml of Kibana server and config.properties in API Gateway

4

Error while running sgtlstool.bat or enable_ssl.bat

The system cannot find the path specified.

It means the JAVA_HOME was not set.

configuring JAVA_HOME will resolve the issue.

References

Downloadable artifacts

Learn more

search-guard-tlstool-1.5.zip (10.7 MB)

Certificates.yml (3.51 KB)

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

image.png

kibanaconfig.PNG