webMethods API Gateway tutorial
Author: Madharsha, Sadiq (sadm@softwareag.com)
Supported Versions: 10.3 Fix 4 and 10.4
Overview of the tutorial
This tutorial explains in detail the steps needed for migrating an API Gateway cluster using Backup mode.
Required knowledge
The tutorial assumes that the reader has,
- a basic knowledge on the API Gateway as a product
- a basic understanding on Elasticsearch (Internal Data Store)
Why?
In earlier versions of API Gateway i.e before 10.3 Fix 4 the migration commands were more complex which involves a multi step process. A lot of manual steps were needed to perform the migration and adding an overhead of restarting API Gateway server and Elasticsearch multiple times. In addition to that the user has to face few more struggles, some of them are, the migration didn't support migration of data from externally configured Elasticsearch and for any troubleshooting, the user has to look out multiple locations for the logs instead of single unified location which is far more easier.
The new migration utility introduced in API Gateway 10.3 Fix 4 will resolve the below issues.
- The commands are very simple and few
- Restart of Elasticsearch and API Gateway server is eliminated
- Migration of Elasticsearch and IS can be done separately
- Migration of data from externally configured Elasticsearch is supported
- Logs all the details to a standard file, migrationLog.txt, a single file for all the log data
- Supports reverting in case of failure in Elasticsearch migration
Prerequisite steps
Complete the below prerequisites to make you ready to get into the details of the staging and promotion in API Gateway.
- Install source and target API Gateway instances. The version of target API Gateway should be higher than source API Gateway. Supported source API Gateway versions are 10.1 and above
- If custom keystore files are used in the source API Gateway installation, copy the files to the same location in the target installation
Details
In this section we will go through the steps for migrating an API Gateway cluster to a higher version using backup mode. The steps are given below.
Step 1: Start source Elasticsearch instances
Go to <SOURCE>\InternalDataStore\config and configure path.repo property in elasticsearch.yml file. Do this for all the nodes. Note that the path.repo should be a shared network folder and should be accessible for all the Elasticsearch nodes in the cluster.
Then start all the <SOURCE> Elasticsearch instances.
Step 2: Take source data backup
Now lets take the backup of source API Gateway data. Go to any one of the Elasticsearch nodes location <SOURCE>\IntegrationServer\instances\default\packages\WmAPIGateway\cli\bin and run the below command.
$> apigw-upgrade-backup.bat -backupDestinationDirectory <path_to_data_store_backup_folder> -backupFileName <backup_file_name_without_spaces_for_the_zip>
Parameter |
Description |
Sample command |
---|---|---|
backupDestinationDirectory |
Same network. The location where the cumulative backup data will get stored. Make sure the directory is already created before executing the command. |
-backupDestinationDirectory C:\migration\backupFolder |
backupFileName |
Backup file name. The name of the .zip file that will be created as part of the backup command execution. Make sure the backup file name is specified in lower case |
-backupFileName backupzipFile |
For Example,
$> apigw-upgrade-backup.bat -backupDestinationDirectory C:\migration\backupfolder -backupFileName backupzipfile
After the command is run successfully, the backup directory would contain the Elasticsearch data folder which is stored with the tenant name (for our use case it is default) and API Gateway configuration data as a zip file (for our use case it is backupzipfile.zip) as seen in the below screenshot. The Elasticsearch data folder is copied from the location we configured in Step 1 by the command.
Step 3: Start target Elasticsearch instance
- After the backup command is executed and the backup is done, stop all the <SOURCE> Elasticsearch nodes.
- Now go to <TARGET_ELASTIC_SEARCH>\config and configure path.repo property in elasticsearch.yml file for all the <TARGET> Elasticsearch nodes. Start any one of the <TARGET> Elasticsearch nodes.
Step 4: Create snapshot in target Elasticsearch instance
When the <TARGET_ELASTIC_SEARCH> is up and running, invoke the below RESP API to create a repository in the <TARGET> machine with tenant name.
PUT /_snapshot/tenant_name
{ "type": "fs", "settings": { "location": <tenant_name> } }
For e.g if we want to create 'a repo for 'default' as tenant name, then the command would be as below.
PUT http://localhost:9240/_snapshot/default
{ "type": "fs", "settings": { "location": "default" } }
After the REST API invocation, a folder with the tenant name (in our use case it is default) will be created under the path.repo folder.
Step 5: Copy Elasticsearch backup data to target Elasticsearch snapshot
This is a preparation step for Elasticsearch data migration. Go to backupDestinationDirectory/<tenant_name> directory configured in Step 2 and copy the contents from Elasticsearch data folder (with tenant name) to the shared repository folder created in Step 4.
For e.g the below is the Elasticsearch data folder for our use case.
Step 6: Run migration for Elasticsearch data
Note: Before running the migration, setup the target Elasticsearch cluster and make sure that all the source and target Elasticsearch instances in the clusters are up and running. But API Gateway instances should not be started on any of the nodes.
Once the above prerequisites are ready we are now ready to run the migration commands to migrate the Elasticsearch data and the API Gateway configurations. In this step we will migrate the Elasticsearch data.
Go to <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate directory in any one of the cluster nodes and run the below command.
$> migrate.bat datastore
After command is run the data store data will be migrated.
Step 7: Run migration for IS configurations
After successful migration of Elasticsearch data, run API Gateway configurations migration using the below command.
$> migrate.bat apigateway -srcFile <backupLoc> fileName.zip -instanceName <old instance name> -newInstanceName <new instance name>
Parameter |
Description |
Sample command |
---|---|---|
srcFile |
Provide the .zip file location that were created as part of backup process. This can also be a shared network file. |
-srcFile C:\migration\backupfolder\backupzipfile.zip |
instanceName |
This is optional parameter. Here we need to pass <SOURCE> instance name . If you don't provide any name then 'default' will be assigned. If you want to migrate different instance other than default, please provide its. (To know more about Integration server instances please refer its doc) |
-instanceName default -instanceName test |
newInstanceName |
This is optional parameter. Here we need to pass <TARGET> instance name . If you don't provide any name then 'default' will be assigned. If you have created a new instance other than default in Integration server and you want to migrate to the new instance then please provide its name for -newInstanceName option |
-newInstanceName default -newInstanceName prod |
For e.g,
$> migrate.bat apigateway -srcFile C:\migration\backupfolder\backupzipfile.zip -instanceName default -newInstanceName default
Now the API Gateway configurations are migrated. This should be done for all the API Gateway nodes in the cluster.
Step 8: Post migration steps
This is a post migration step. Follow the below steps.
- Shutdown the <TARGET> Elasticsearch instances
- (Optional) If Elasticsearch nodes are configured externally to API Gateway nodes, Start all those externally configured Elasticsearch nodes to setup the cluster
- Start the <TARGET> API Gateway nodes
General Migration configuration
API Gateway customers can modify certain parameters for migration based on their requirements by modifying the property file migration.properties which is located at <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate. This property file is instance specific and customers can have different configurations while migrating different instances.
Property |
Description |
Default value |
Possible values |
---|---|---|---|
apigateway.migration.srcTenantName |
By default, the source tenant is assumed as default. But If the source API Gateway has multiple tenants, this property can be used to specify the tenant name from which the data has to be migrated to the target tenant. |
default |
Any available tenant in Elasticsearch |
apigateway.migration.batchSize |
The batch size with which the data is processed. For e.g if size is 100 then by default 100 documents will be processed first. If the network is slow we can decrease this value and if the network is better we can increase this value. |
100 |
Appropriate batch size. It depends on the number of documents and the size of the documents in the data store |
apigateway.migration.logLevel |
Log level for migration. we can change log level to debug, error etc. |
info |
info,debug,error,warn,trace |
apigateway.migration.reindex.status.check.sleep.interval |
Interval configuration in milliseconds. Once the re-indexing process has started from source to target instances, migration process will wake up after every configured sleep interval to check whether the re-indexing is complete. It will check the status of the task id |
5000 |
Appropriate sleep interval |
Recovery
During migration, if there is any problem in the execution or any of the handlers got failed, to make sure that assets are migrated properly, we can clean the target instance once and then re run the migration. This clean command will clean the target data store (the one configured in the config.properties of target machine) . During this procedure all the indices will be removed and this is a non reversible action. Before cleaning the data we will also take a backup of the existing data (you can also restore it). Once cleaned, we can re-run the migration. Also once you trigger the clean command, this process will wait for 5 seconds and if you wrongly triggered it and you want to kill this process you can do that with in that 5 seconds interval.
Clean command
Go to <TARGET>\IntegrationServer\instances\default\packages\WmAPIGateway\bin\migrate and run the below command.
$> migrate.bat clean
Troubleshooting
# |
Issue Description |
Scenario name |
Reason |
Remedy |
---|---|---|---|---|
1 |
ElasticsearchException[Error while reindexing APIs. Error type - illegal_argument_exception, reason [localhost:1240] not whitelisted in reindex.remote.whitelist] |
Direct - Standalone |
Remote re-indexing property missed in target elasticsearch.yml |
Add the below property in <TARGET> elasticsearch.yml reindex.remote.whitelist: <sourcehost>:<sourcehttpport> |
2 |
Exception thrown during migration operation. Exiting the operation. No Such command - -srcDir |
Direct - Standalone |
Wrong command usage |
datastore/apigateway argument must be passed migration.bat apigateway -srcDir C:\SoftwareAG_10.1 -instanceName default -newInstanceName default |
3 |
Deleting the backup folder before migration... Migration Process 0% [>] 0/100 (0:00:00) java.io.FileNotFoundException: 15:41:13.580 [main] ERROR com.softwareag.apigateway.utility.command.backup.instance.BackupApiGatewayInstance - Error occurred while trying to parse the Manifest file to obtain the version information. java.io.FileNotFoundException:(The system cannot find the file specified) |
Backup - Standalone |
Existence of isExtract file due to migration failure of previous steps |
Delete the isExtract folder in C:\<<InstallationDir>>\ |
4 |
elasticsearchclient bean creation exception |
All |
Make sure the respective Elasticsearch nodes is running |
|
5 |
[2018-12-19T12:34:03,892][WARN ][o.e.r.VerifyNodeRepositoryAction] [SAG-2KXGBH2.eur.ad.sag1544697848038] [default] failed to verify repository org.elasticsearch.repositories.RepositoryVerificationException: [default] a file written by master to the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] cannot be accessed on the node [{SAG-2KXGBH2.eur.ad.sag1544697848038} {yL84xqhZQSuFfUmj0GrFbQ}{uGv-BmTBT6e8LCpOsxpGOg} {10.60.37.18}{10.60.37.18:9340}]. This might indicate that the store [C:\SoftwareAG_10.3\InternalDataStore\dasoSnap\default] is not shared between this node and the master node or that permissions on the store don't allow reading files written by the master node |
Backup Mode - Cluster |
Make sure path.repo is a single common location reachable and accessible by all other nodes |
Configure single common shared path in elasticsearch.yml (path.repo) |
6 |
The system cannot find the path specified |
All |
Make sure that the bat from the proper location |
Make sure that you are running the bat from the proper location |
7 |
2019-10-16 11:59:36 ERROR ESDataStoreHandler:327 - {"type":"illegal_argument_exception","reason":"Remote responded with a chunk that was too large. Use a smaller batch size.","caused_by":{"type":"content_too_long_exception","reason":"entity content is too long [185463385] for the configured buffer limit [104857600]"}} |
All |
The size of the documents selected for reindexing as per the batch size configuration is large |
Try with a smaller batch size number. |
8 | Error while getting task details for taks id - 6L6RMNEOQF64lQbNeI7J_g:683354. Message - task [6L6RMNEOQF64lQbNeI7J_g:683354] isn't running and hasn't stored its results | All | When the space in machine is less than 10% , elasticsearch marks index as readonly hence reindex task fails abruptly | Increase the memory |
References
- https://www.elastic.co/guide/en/elasticsearch/reference/current/index.html
- https://www.elastic.co/guide/en/elasticsearch/reference/master/docs-reindex.html
Learn more
- For backup and restore refer http://techcommunity.softwareag.com/pwiki/-/wiki/Main/Back+up+and+restore+of+API+Gateway+assets