Securing Elasticsearch for API Gateway 10.5

Clench_Paign_James3 · April 5, 2020, 8:18am

webMethods API Gateway tutorial

Author: Rizwan, Mohammed (mriz@softwareag.com)
Supported Versions: 10.5

Overview of the tutorial

This tutorial helps to understand how the InternalDataStore (or a simple Elasticsearch instance) can be secured using Search Guard, an Elasticsearch plugin that offers encryption, authentication and authorization. In this context, this plugin helps to secure the InternalDataStore by exposing it over HTTPS protocol and enabling a basic authentication by configuring users.

This tutorial covers the below steps in detail.

Installation and initialization of Search Guard plugin
Adding users for basic authentication
Changing Kibana configurations to securely connect to Elasticsearch
Configuring API Gateway to securely connect to Elasticsearch
Configuring Search Guard with user generated certificates

Required knowledge

The reader has to have:

a basic understanding of API Gateway and its communication with InternalDataStore for storing data
a basic understanding of Kibana and its communication with InternalDataStore for rendering the dashboards in API Gateway

Why?

InternalDataStore, a wrapper over Elasticsearch, being the back-end store for API Gateway, it is necessary to protect this store from attackers and other misuses which would eventually damage the entire functionality of the API Gateway in a way that it can not be recoverable. Hence it is almost important to protect the InternalDataStore and this security aspect can be achieved by using Search Guard, a third party plugin, which can be plugged into the Elasticsearch server for providing security functionalities.

Prerequisite steps

Complete the below prerequisites steps before get into the details of securing InternalDataStore of API Gateway using SearchGuard plugin.

Install API Gateway advanced edition of version 10.2 and above

Details

The below sections describe the details of securing InternalDataStore using Search Guard plugin by installing the plugin and adding the necessary configurations both in InternalDataStore, Kibana and API Gateway.

Step 1: Installation and initialization of Search Guard plugin

This section explains how to install the Search Guard plugin and make the InternalDataStore ready for enabling security.

Shutdown the API Gateway. Open <SAG_Root>/InternalDataStore /bin/enable_ssl.bat and comment out the last line ..\plugins\search-guard-5\tools\sgadmin.bat and save.

Copy sagconfig from <SAG_Root>/IntegrationServer/instances/<Instance_Name>/packages/WmAPIGateway/config/resources/elasticsearch to <SAG_Root>/InternalDataStore. Then copy the certificates node-0-keystore.jks and truststore.jks from <SAG_Root>/InternalDataStore/sagconfig to <SAG_Root>/InternalDataStore/config.

Open <SAG_Root>/InternalDataStore/config/elasticsearch.yml file. Remove all the properties that start with "searchguard" if such present and add the Search Guard properties as given below and save the file.

Search Guard properties

searchguard.ssl.transport.keystore_type: JKS
searchguard.ssl.transport.keystore_filepath: node-0-keystore.jks
searchguard.ssl.transport.keystore_alias: cn=node-0
searchguard.ssl.transport.keystore_password: a362fbcce236eb098973
searchguard.ssl.transport.truststore_type: JKS
searchguard.ssl.transport.truststore_filepath: truststore.jks
searchguard.ssl.transport.truststore_alias: root-ca-chain
searchguard.ssl.transport.truststore_password: 2c0820e69e7dd5356576
searchguard.ssl.transport.enforce_hostname_verification: false
searchguard.ssl.transport.resolve_hostname: false
searchguard.ssl.transport.enable_openssl_if_available: true

searchguard.ssl.http.enabled: true
searchguard.ssl.http.keystore_type: JKS
searchguard.ssl.http.keystore_filepath: node-0-keystore.jks
searchguard.ssl.http.keystore_alias: cn=node-0
searchguard.ssl.http.keystore_password: a362fbcce236eb098973
searchguard.ssl.http.truststore_type: JKS
searchguard.ssl.http.truststore_filepath: truststore.jks
searchguard.ssl.http.truststore_alias: root-ca-chain
searchguard.ssl.http.truststore_password: 2c0820e69e7dd5356576
searchguard.ssl.http.clientauth_mode: OPTIONAL

searchguard.enable_snapshot_restore_privilege: true

searchguard.check_snapshot_restore_write_privileges: true

searchguard.restapi.roles_enabled: ["SGS_ALL_ACCESS"]

searchguard.authcz.admin_dn:
- "CN=sgadmin"

Search Guard properties

A brief description of each Search Guard property is given below.

Item	Desc	Possible Values	Default
TRANSPORT ( 2-Way authentication is enabled by default)
searchguard.ssl.transport.keystore_type	Type of keystore	JKS, PKCS12	JKS
searchguard.ssl.transport.keystore_filepath	Location where the keystore
searchguard.ssl.transport.keystore_alias	Keystore entry name if there are more than one entries.
searchguard.ssl.transport.keystore_password	Password to access keystore.
searchguard.ssl.transport.truststore_type	Type of truststore	JKS, PKCS12	JKS
searchguard.ssl.transport.truststore_filepath	Location where the truststore
searchguard.ssl.transport.truststore_alias	Truststore entry name if there are more than one entries.
searchguard.ssl.transport.truststore_password	Password to access truststore.
searchguard.ssl.transport.enforce_hostname_verification	If true, the hostname mentioned in certificate will be validated. We set this to false as ours is general purpose self signed certs.	true / false	true
searchguard.ssl.transport.resolve_hostname	Applicable only if above property is true. If true, the hostname will be resolved against the DNS server. We set this to false as ours is general purpose self signed certs.	true / false	true
searchguard.ssl.transport.enable_openssl_if_available	Use if OpenSSL is available instead of JDK SSL.	true / false	true
HTTP
searchguard.ssl.http.enabled	To enable the SSL for REST interface ( HTTP). We set this to true.	true / false	false
searchguard.ssl.http.keystore_type	Type of keystore	JKS, PKCS12	JKS
searchguard.ssl.http.keystore_filepath	Location where the keystore
searchguard.ssl.http.keystore_alias	Keystore entry name if there are more than one entries.
searchguard.ssl.http.keystore_password	Password to access keystore.
searchguard.ssl.http.truststore_type	Type of truststore	JKS, PKCS12	JKS
searchguard.ssl.http.truststore_filepath	Location where the truststore
searchguard.ssl.http.truststore_alias	Truststore entry name if there are more than one entries
searchguard.ssl.http.truststore_password	Password to access truststore.
searchguard.ssl.http.clientauth_mode	Option to enable 2-way authentication. REQUIRE : Client will be challenged for client certificate. OPTIONAL : Will be used if client certificate is available NONE : Ignore client certificate even if it's available.	REQUIRE, OPTIONAL, NONE	OPTIONAL
Search Guard Admin
searchguard.authcz.admin_dn	Search Guard maintains all the data in a index called "searchguard". This is accessible to only users ( client cert which will be passed in sdadmin command) configured here.
Misc
searchguard.cert.oid	All certificates used by the nodes on transport level need to have the `oid` field set to a specific value.This oid value is checked by Search Guard to identify if an incoming request comes from a trusted node in the cluster or not. In the former case, all actions are allowed, in the latter case, privilege checks apply. Plus, the oid is also checked whenever a node wants to join the cluster.		'1.2.3.4.5.5'
searchguard.config_index_name	Index where all the security configuration is stored. It's not confgurable for now but in the future		searchguard
searchguard.enable_snapshot_restore_privilege searchguard.check_snapshot_restore_write_privileges	To perform snapshot and restore operations, a user needs to have special privileges assigned. These two lines enable these privileges
searchguard.restapi.roles_enabled	Tells Search Guard which Search Guard roles can access the REST Management API to perform changes to the configuration

Now run <SAG_Root>/InternalDataStore/bin/enable_ssl.bat file. This will Install Search Guard plugin and start the InternalDataStore.

Now shutdown and restart the InternalDataStore. Go to <SAG_Root>/InternalDataStore/plugins/search-guard-7/tools and execute the below command.

sgadmin.bat -cd ..\..\..\sagconfig\ -ks ..\..\..\sagconfig\sgadmin-keystore.jks -kspass 49fc2492ebbcfa7cfc5e -ts ..\..\..\sagconfig\truststore.jks -tspass 2c0820e69e7dd5356576 -nhnv -p 9340 -cn SAG_EventDataStore

This will Initialize the InternalDataStore. Refer the image given below.

Note: The Password -kspass 49fc2492ebbcfa7cfc5e and –tspass 2c0820e69e7dd5356576 are default passwords for keystore and truststore and the default certificates are not safe for production.

Step 2: Adding users for basic authentication

In this section let us see how an user can be provisioned in the Search Guard plugin to provide a http basic authentication.

Go to <SAG_Root>\InternalDataStore\sagconfig and open sg_roles_mapping.yml file. Add the username (e.g. TestUser) in the users list as given in the below image.

Before providing the password it must be converted into hash code. To generate the hash code for your password , execute <SAG_Root>\InternalDataStore\plugins\search-guard-7\tools\hash.bat. Type the password, press enter and it will generate the hash code as given in below image.

Now open sg_internal_users.yml file from <SAG_Root>\InternalDataStore\sagconfig and add the username and password as given in the below image.

After completing the configurations, initialize Search Guard plugin again by executing the sgadmin.bat command as we did before. After initialization shutdown and restart the InternalDataStore. Now the InternalDataStore is running securely by opening a HTTPS port and requesting basic authentication details.

Step 3: Changing Kibana configurations to connect to Elasticsearch

Now we need to modify the Kibana server configurations to connect to the Elasticsearch securely over HTTPS port using basic authentication details.

Open kibana.yml file from the directory <SAG-Home>\profiles\IS_default\apigateway\dashboard\config
Remove the comments and update the following properties given below
- elasticsearch.username: TestUser
- elasticsearch.password: TestUser@123
- elasticsearch.ssl.verificationMode: certificate
- elasticsearch.ssl.certificateAuthorities: <file path of your root-ca.pem certificate>
- elasticsearch.url: https://<hostname>:9240
After changing these configurations open uiconfiguration.properties file at <SAG_Home>\profiles\IS_default\apigateway\config and set apigw.kibana.autostart to false

Sample kibana.yml file image is given below.

Step 4: Changing API Gateway configurations to connect to Elasticsearch

Finally change the API Gateway configurations to connect to InternalDataStore securely. Follow the below steps.

Go to <SAG_HOME>/IntegrationServer/instances/default/packages/WmAPIGateway/config/resources/elasticsearch and open config.properties file. Uncomment the following properties and provide the values for them as below
- pg.gateway.elasticsearch.http.username=TestUser
- pg.gateway.elasticsearch.http.password=TestUser@123
- pg.gateway.elasticsearch.https.truststore.filepath=C:/SoftwareAG-A-105/InternalDataStore/sagconfig/truststore.jks
- pg.gateway.elasticsearch.https.truststore.password=2c0820e69e7dd5356576
- pg.gateway.elasticsearch.https.enabled=true
After making all the configurations, start the InternalDataStore manually
When InternalDataStore is up and running, start the Kibana server manually (To start kibana server manually run kibana.bat file at <SAG_HOME>/profiles\IS_default\apigateway\dashboard\bin)
Now start the API Gateway and you would be able to login and create APIs. Analytics page should be accessible without any challenge window for user credentials

Step 5: Configuring Search Guard with user generated certificates (optional)

The API Provider can generate his own certificates to be used with Search Guard instead of the default certificates that are shipped with API Gateway. To achieve this, Search Guard provides an offline TLS tool, using which all the required certificates can be generated for running Search Guard in a production environment. Follow the below steps to generate certificates using Search Guard offline TLS tool.

Download the tool zip file from https://search.maven.org/#search%7Cga%7C1%7Ca%3A%22search-guard-tlstool%22 (the same can be found in the attachments) and unzip it to a directory. Create a yaml file at <OfflineTool Installation Directory>\config. On executing the TLS tool command, it will read the node and certificate configuration settings from this yaml file and outputs the generated files in a configured directory. The sample file is given below. Please find the attachments for the sample file, Certificates.yml.

After configuring the yaml file run the below command to generate the required certificates.
<OfflineTool Installation Directory>/tools/sgtlstool.bat -c ../config/Demo.yml -ca -crt
The generated certificates can be outputted at <OfflineTool Installation Directory>/tools/out

For further details on this refer the link provided in the References. It is mandatory to copy the certificates listed below from <OfflineTool Installation Directory>/tools/out to <SAG_Root>/InternalDataStore/config folder.

Dinesh-node-1.key
Dinesh-node-1.pem
Dinesh-node-1_http.pem
Dinesh-node-1_http.key
Dinesh-client.pem
Dinesh-client.key
root-ca.pem
root-ca.key

Now configure the generated certificates in the InternalDataStore elasticsearch.yml file.

After completing all the above steps start the InternalDataStore manually. After InternalDataStore is up, since the Search Guard is not initialized with the latest certificates, a log message would be shown saying that the Search Guard is not initialized. The following section provide the steps to initialize the Search Guard with the generated certificates.

Open command prompt and change directory to <SAG-Home>\InternalDataStore \plugins\search-guard-5\tools
Execute the command sgadmin.sh -cd ..\sagconfig –nhnv –icl –cacert ..\..\..\config\root-ca.pem -cert ..\..\..\config\Dinesh-client.pem -key ..\..\..\config\Dinesh-client.key -keypass <your certificate password> -p 9340 . A log "Done with success" will show up
Now shutdown and restart the InternalDataStore. The InternalDataStore will now use the generated certificates for SSL communication

Additional Steps while using user generated certificates

Using a key tool import generated pem certificate "root-ca.pem" into truststore.
Steps for generating a empty truststore:
- keytool -genkey -keyalg RSA -alias endeca -keystore truststore.jks
- keytool -delete -alias endeca -keystore truststore.jks
Command for importing pem file into truststore
1. keytool -import -v -trustcacerts -alias endeca-ca -file root-ca.pem -keystore truststore.jks

After which, use the generated truststore in step 3.

Troubleshooting

#	Problem	Solution
1	Search Guard may not be initialized. Initialize Search Guard Using sgadmin tool	Make sure that Search Guard is initialized properly using the steps given above
2	ES-Log contains entries like the following: [2020-01-01T12:00:00,097] [INFO] [c.f.s.c.ConfigurationRepository] Search Guard License Type: TRIAL, valid [2020-01-01T12:00:00,097] [WARN] [c.f.s.c.ConfigurationRepository] Your Search Guard license expires in 17 days.	The Community Edition does not require a license. To use it, just install the Enterprise Edition and disable all commercial features by adding the following line to `elasticsearch.yml` and restart the node: `searchguard.enterprise_modules_enabled: false` Setting this flag will disable any commercial module or custom authentication domain and will run only the free Community features of Search Guard.
3	Error in ElasticSearch: Speaks http plaintext instead of SSL, will close the channel	It means Kibana or API Gateway is still communicating with Elasticsearch in HTTP. Configure the https protocol and certificates in kibana.yml of Kibana server and config.properties in API Gateway
4	Error while running sgtlstool.bat or enable_ssl.bat The system cannot find the path specified.	It means the JAVA_HOME was not set. configuring JAVA_HOME will resolve the issue.