Migrating API Gateway cluster using Backup mode

Clench_Paign_James3 · February 2, 2021, 10:00am

webMethods API Gateway tutorial

Author: Madharsha, Sadiq (sadm@softwareag.com)
Supported Versions: 10.3 Fix 4 and 10.4

Note: For migrating to API Gateway 10.5 using Backup mode for cluster please refer Migrating to API Gateway 10.5 using Backup mode for cluster.

Overview of the tutorial

This tutorial explains in detail the steps needed for migrating an API Gateway cluster using Backup mode.

Note: This tutorial is applicable for on-premise installation only

Required knowledge

The tutorial assumes that the reader has,

a basic knowledge on the API Gateway as a product
a basic understanding on Elasticsearch (Internal Data Store)

Why?

In earlier versions of API Gateway i.e before 10.3 Fix 4 the migration commands were more complex which involves a multi step process. A lot of manual steps were needed to perform the migration and adding an overhead of restarting API Gateway server and Elasticsearch multiple times. In addition to that the user has to face few more struggles, some of them are, the migration didn't support migration of data from externally configured Elasticsearch and for any troubleshooting, the user has to look out multiple locations for the logs instead of single unified location which is far more easier.

The new migration utility introduced in API Gateway 10.3 Fix 4 will resolve the below issues.

The commands are very simple and few
Restart of Elasticsearch and API Gateway server is eliminated
Migration of Elasticsearch and IS can be done separately
Migration of data from externally configured Elasticsearch is supported
Logs all the details to a standard file, migrationLog.txt, a single file for all the log data
Supports reverting in case of failure in Elasticsearch migration

Prerequisite steps

Complete the below prerequisites to make you ready to get into the details of the staging and promotion in API Gateway.

Install source and target API Gateway instances. The version of target API Gateway should be higher than source API Gateway. Supported source API Gateway versions are 10.1 and above
If custom keystore files are used in the source API Gateway installation, copy the files to the same location in the target installation

Details

In this section we will go through the steps for migrating an API Gateway cluster to a higher version using backup mode. The steps are given below.

Note: In APIGateway 10.2 and above the the folder name EventDataStore has been changed to InternalDataStore. Throughout this tutorial we refer source API Gateway installation directory as <SOURCE>, target API Gateway installation directory as <TARGET> and target elasticsearch is <TARGET_ELASTIC_SEARCH>

Step 1: Start source Elasticsearch instances

Go to <SOURCE>\InternalDataStore\config and configure path.repo property in elasticsearch.yml file. Do this for all the nodes. Note that the path.repo should be a shared network folder and should be accessible for all the Elasticsearch nodes in the cluster.

Then start all the <SOURCE> Elasticsearch instances.

Step 2: Take source data backup

Now lets take the backup of source API Gateway data. Go to any one of the Elasticsearch nodes location <SOURCE>\IntegrationServer\instances\default\packages\WmAPIGateway\cli\bin and run the below command.

$> apigw-upgrade-backup.bat -backupDestinationDirectory <path_to_data_store_backup_folder> -backupFileName <backup_file_name_without_spaces_for_the_zip>

Parameter	Description	Sample command
backupDestinationDirectory	Same network. The location where the cumulative backup data will get stored. Make sure the directory is already created before executing the command.	-backupDestinationDirectory C:\migration\backupFolder
backupFileName	Backup file name. The name of the .zip file that will be created as part of the backup command execution. Make sure the backup file name is specified in lower case	-backupFileName backupzipFile

For Example,

$> apigw-upgrade-backup.bat -backupDestinationDirectory C:\migration\backupfolder -backupFileName backupzipfile

Note: Make sure the backup destination directory is already created and the backup file name be specified in lower case.

After the command is run successfully, the backup directory would contain the Elasticsearch data folder which is stored with the tenant name (for our use case it is default) and API Gateway configuration data as a zip file (for our use case it is backupzipfile.zip) as seen in the below screenshot. The Elasticsearch data folder is copied from the location we configured in Step 1 by the command.

Step 3: Start target Elasticsearch instance

After the backup command is executed and the backup is done, stop all the <SOURCE> Elasticsearch nodes.
Now go to <TARGET_ELASTIC_SEARCH>\config and configure path.repo property in elasticsearch.yml file for all the <TARGET> Elasticsearch nodes. Start any one of the <TARGET> Elasticsearch nodes.

Step 4: Create snapshot in target Elasticsearch instance

When the <TARGET_ELASTIC_SEARCH> is up and running, invoke the below RESP API to create a repository in the <TARGET> machine with tenant name.

PUT /_snapshot/tenant_name

{ "type": "fs", "settings": { "location": <tenant_name> } }

For e.g if we want to create 'a repo for 'default' as tenant name, then the command would be as below.

PUT http://localhost:9240/_snapshot/default

{ "type": "fs", "settings": { "location": "default" } }

After the REST API invocation, a folder with the tenant name (in our use case it is default) will be created under the path.repo folder.

Step 5: Copy Elasticsearch backup data to target Elasticsearch snapshot

This is a preparation step for Elasticsearch data migration. Go to backupDestinationDirectory/<tenant_name> directory configured in Step 2 and copy the contents from Elasticsearch data folder (with tenant name) to the shared repository folder created in Step 4.

For e.g the below is the Elasticsearch data folder for our use case.

Step 6: Run migration for Elasticsearch data

Note: Before running the migration, setup the target Elasticsearch cluster and make sure that all the source and target Elasticsearch instances in the clusters are up and running. But API Gateway instances should not be started on any of the nodes.

Once the above prerequisites are ready we are now ready to run the migration commands to migrate the Elasticsearch data and the API Gateway configurations. In this step we will migrate the Elasticsearch data.

Go to <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate directory in any one of the cluster nodes and run the below command.

$> migrate.bat datastore

After command is run the data store data will be migrated.

Step 7: Run migration for IS configurations

After successful migration of Elasticsearch data, run API Gateway configurations migration using the below command.

$> migrate.bat apigateway -srcFile <backupLoc> fileName.zip -instanceName <old instance name> -newInstanceName <new instance name>

Parameter

Description

Sample command

srcFile

Provide the .zip file location that were created as part of backup process. This can also be a shared network file.

-srcFile C:\migration\backupfolder\backupzipfile.zip

instanceName

This is optional parameter. Here we need to pass <SOURCE> instance name .

If you don't provide any name then 'default' will be assigned. If you want to migrate different instance other than default, please provide its. (To know more about Integration server instances please refer its doc)

-instanceName default

-instanceName test

newInstanceName

This is optional parameter. Here we need to pass <TARGET> instance name .

If you don't provide any name then 'default' will be assigned. If you have created a new instance other than default in Integration server and you want to migrate to the new instance then please provide its name for -newInstanceName option

-newInstanceName default

-newInstanceName prod

For e.g,

$> migrate.bat apigateway -srcFile C:\migration\backupfolder\backupzipfile.zip -instanceName default -newInstanceName default

Now the API Gateway configurations are migrated. This should be done for all the API Gateway nodes in the cluster.

Step 8: Post migration steps

This is a post migration step. Follow the below steps.

Shutdown the <TARGET> Elasticsearch instances
(Optional) If Elasticsearch nodes are configured externally to API Gateway nodes, Start all those externally configured Elasticsearch nodes to setup the cluster
Start the <TARGET> API Gateway nodes

General Migration configuration

API Gateway customers can modify certain parameters for migration based on their requirements by modifying the property file migration.properties which is located at <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate. This property file is instance specific and customers can have different configurations while migrating different instances.

Property	Description	Default value	Possible values
apigateway.migration.srcTenantName	By default, the source tenant is assumed as default. But If the source API Gateway has multiple tenants, this property can be used to specify the tenant name from which the data has to be migrated to the target tenant.	default	Any available tenant in Elasticsearch
apigateway.migration.batchSize	The batch size with which the data is processed. For e.g if size is 100 then by default 100 documents will be processed first. If the network is slow we can decrease this value and if the network is better we can increase this value.	100	Appropriate batch size. It depends on the number of documents and the size of the documents in the data store
apigateway.migration.logLevel	Log level for migration. we can change log level to debug, error etc.	info	info,debug,error,warn,trace
apigateway.migration.reindex.status.check.sleep.interval	Interval configuration in milliseconds. Once the re-indexing process has started from source to target instances, migration process will wake up after every configured sleep interval to check whether the re-indexing is complete. It will check the status of the task id	5000	Appropriate sleep interval

Recovery

During migration, if there is any problem in the execution or any of the handlers got failed, to make sure that assets are migrated properly, we can clean the target instance once and then re run the migration. This clean command will clean the target data store (the one configured in the config.properties of target machine) . During this procedure all the indices will be removed and this is a non reversible action. Before cleaning the data we will also take a backup of the existing data (you can also restore it). Once cleaned, we can re-run the migration. Also once you trigger the clean command, this process will wait for 5 seconds and if you wrongly triggered it and you want to kill this process you can do that with in that 5 seconds interval.

Clean command

Note (Before running clean command): If the <TARGET> is 10.5 and the clean command is executed in a cluster, go to <SOURCE>\InternalDataStore\config and configure path.repo property in elasticsearch.yml file for all the nodes. Make sure that the path.repo is a shared network folder and should be accessible for all the Elasticsearch nodes in the cluster.

Go to <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate and run the below command.

$> migrate.bat clean

Note: Since this command removes all the assets from data store, make sure that the target data store is properly configured in config.properties which is located at <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\config\resources\elasticsearch.

Troubleshooting

#	Issue Description	Scenario name	Reason	Remedy
1	ElasticsearchException[Error while reindexing APIs. Error type - illegal_argument_exception, reason [localhost:1240] not whitelisted in reindex.remote.whitelist]	Direct - Standalone	Remote re-indexing property missed in target elasticsearch.yml	Add the below property in <TARGET> elasticsearch.yml reindex.remote.whitelist: <sourcehost>:<sourcehttpport>
2	Exception thrown during migration operation. Exiting the operation. No Such command - -srcDir	Direct - Standalone	Wrong command usage	datastore/apigateway argument must be passed migration.bat apigateway -srcDir C:\SoftwareAG_10.1 -instanceName default -newInstanceName default
3	Deleting the backup folder before migration... Migration Process 0% [>] 0/100 (0:00:00) java.io.FileNotFoundException: 15:41:13.580 [main] ERROR com.softwareag.apigateway.utility.command.backup.instance.BackupApiGatewayInstance - Error occurred while trying to parse the Manifest file to obtain the version information. java.io.FileNotFoundException:(The system cannot find the file specified)	Backup - Standalone	Existence of isExtract file due to migration failure of previous steps	Delete the isExtract folder in C:\<<InstallationDir>>\
4	elasticsearchclient bean creation exception	All	Make sure the respective Elasticsearch nodes is running
5	[2018-12-19T12:34:03,892][WARN ][o.e.r.VerifyNodeRepositoryAction] [SAG-2KXGBH2.eur.ad.sag1544697848038] [default] failed to verify repository org.elasticsearch.repositories.RepositoryVerificationException: [default] a file written by master to the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] cannot be accessed on the node [{SAG-2KXGBH2.eur.ad.sag1544697848038} {yL84xqhZQSuFfUmj0GrFbQ}{uGv-BmTBT6e8LCpOsxpGOg} {10.60.37.18}{10.60.37.18:9340}]. This might indicate that the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node	Backup Mode - Cluster	Make sure path.repo is a single common location reachable and accessible by all other nodes	Configure single common shared path in elasticsearch.yml (path.repo)
6	The system cannot find the path specified	All	Make sure that the bat from the proper location	Make sure that you are running the bat from the proper location
7	2019-10-16 11:59:36 ERROR ESDataStoreHandler:327 - {"type":"illegal_argument_exception","reason":"Remote responded with a chunk that was too large. Use a smaller batch size.","caused_by":{"type":"content_too_long_exception","reason":"entity content is too long [185463385] for the configured buffer limit [104857600]"}}	All	The size of the documents selected for reindexing as per the batch size configuration is large	Try with a smaller batch size number.
8	Error while getting task details for taks id - 6L6RMNEOQF64lQbNeI7J_g:683354. Message - task [6L6RMNEOQF64lQbNeI7J_g:683354] isn't running and hasn't stored its results	All	When the space in machine is less than 10% , elasticsearch marks index as readonly hence reindex task fails abruptly	Increase the memory

References

Learn more

For backup and restore refer http://techcommunity.softwareag.com/pwiki/-/wiki/Main/Back+up+and+restore+of+API+Gateway+assets