Author: Madhavan Kidambi Varadan (mki@softwareag.com)
Supported Versions: API Gateway 10.1 and above, Deployer 10.1 and above, Asset Build Environment 10.1 and above
webMethods API Gateway tutorial
Overview of the tutorial
This tutorial explains in detail how the assets in API Gateway like APIs ,Policies, Applications, etc. are moved from one stage of SDLC to another using Asset build environment (ABE) and webmethods Deployer. The automation scripts that are provided by ABE and deployer help to achieve the automation of their promotion. This tutorial ends with an insight on how Continuous Integration and Continuous Delivery (CI/CD) of the assets can be achieved.
We will go through the following steps in detail in this tutorial,
- Building API Gateway assets using ABE
- Promoting the built assets using webmethods Deployer
- DevOps (CI/CD) use case of API Gateway assets
Note
From API Gateway 10.2 Staging and Promotion in API Gateway can be achieved using 2 ways
- Staging & Promotion using webMethods Deployer and webMethods Asset Build Environment (ABE) – More suitable for existing webMethods customers who already leverage Deployer and ABE for promotion of Integration Server assets across different stages
- Staging & Promotion using API Gateway Promotion Management – More suitable for customers who would like to manage stages and promotion from within API Gateway User Interface
Continuous Integration and Continuous Delivery can be achieved using any of the ways.
Required knowledge
- Basic knowledge on the API Gateway like creating APIs and policies
- This tutorial explains the bare minimum details of webMethod Deployer and Asset Build environment required for this tutorial. Hence the reader has to go through the respective products documentation for more fine grained details
Why?
A typical enterprise-level customer separate their solution according to the different phases of the SDLC. The most common pattern is separation by development, QA and production stages. Promotion refers to the act of moving API Gateway artifacts from one stage to another.
As each organization builds APIs using API Gateway for easy consumption and monetization, the continuous integration and delivery are integral part of the API Gateway solutions to get hold of the fast moving market. We need to automate the management of APIs and policies to speed up the deployment, introduce continuous integration concepts and place API artifacts under source code management. As new apps are deployed, the API definitions can change and those changes have to be propagated to other external products like API portal. This requires the API owner to update the associated documentation and in most cases this process is a tedious manual exercise. In order to address this issue, it is a key to bring in DevOps style automation to the API life cycle management process in API Gateway. With this, enterprises can deliver continuous innovation with speed and agility, ensuring that new updates and capabilities are automatically, efficiently and securely delivered to their developers and partners in a timely fashion and without manual intervention. This enables a team of API Gateway policy developers to work in parallel developing APIs and policies to be deployed as a single API Gateway configuration.
Prerequisite steps
Complete the below prerequisites to make you ready to get into the details of the staging and promotion in API Gateway if you are working with on-premise installation.
- Install API Gateway of version 10.1 or above
- Install Asset build environment of version same as API Gateway (ABE can be independently installed on a separate instance)
- Install webMethods Deployer of version same as API Gateway (Deployer can be independently installed on a separate instance)
Details
In the following sections we will go through the Staging, Promotion and DevOps of API Gateway assets in detail.
Staging and Promotion
API Gateway assets are mainly APIs and the related policies , Global policies , applications, packages and plans. Let us now see how the said assets can be staged and promoted from a Source API Gateway to a target API Gateway.
Our promotion process consists of two steps.
- Building API Gateway assets using ABE
- Promoting the built assets using webmethods Deployer
Step 1 : Build API Gateway assets using ABE
In the first step we build all our assets into a repository as a binary form. This build step is done with webMethods Asset build environment (ABE) which is a simple ant based tool . ABE will contain the build script and build properties file that are needed to build the binary repository for all webMethods runtimes which includes API Gateway. On execution of the script, ABE pulls the assets from the source API Gateway as configured in the build properties file. It then converts these assets as binary which is a zip file containing the APIs,Policy definitions and their dependencies.
The API Gateway scripts in ABE provides support for building assets in the following two ways.
- Create a local repository containing the API Gateway assets binary by connecting to the source API Gateway
- Create an repository by either using an local repository which is a mostly a Version Control System like SVN, Github, etc. This will be very useful for DevOps which is described later in detail in this tutorial
API Gateway properties in ABE
ABE contains a master properties file and a build.properties that controls the build settings for the repository based build process. APIGateway assets can be built by executing the build.bat/.sh file which presents in the <SoftwareAG_installation_directory>\common\AssetBuildEnvironment\bin location, by setting the property enable.build.APIGateway to true.
There are various other properties needed for building API Gateway assets. The below table explains them in detail.
| # | Property name | Optional or Mandatory | Default value | What is this meant for? |
|---|---|---|---|---|
| 1 | enable.build.APIGateway Mandatory | Mandatory | False | Only if this parameter is set to true, API Gateway Assets will be built |
| 2 | apigateway.buildLocalRepoOnly | Optional | False |
If this parameter is set to true , A local repo only will be built and not the repo as needed by WmDeployer. This will be greatly used for Devops scenario to compare the Assets using an Version Control system. |
| 3 | build.output.dir | Mandatory | NA |
If apigateway.buildLocalRepoOnly is true this will be the location of the VCS. If apigateway.buildLocalRepoOnly is false then this is the root directory where the output (binary and ACDL) of the build will be placed. Also this path will also be indexed. |
| 4 | apigateway.repo.createFromLocalRepo | Optional | False |
If this parameter is set to true then the build script will use the location set as a part of apigateway.repo.localRepo.path to build the Repo for WmDeployer usage. apigateway.buildLocalRepoOnly will be false when this is true. |
| 5 | apigateway.repo.localRepo.path | Optional, Mandated only when apigateway.repo.createFromLocalRepo is true | NA |
Local path where the repository when apigateway.repo.createFromLocalRepo is true. Mostly points to an VCS. |
| 6 | apigateway.assets.file | Optional, Mandated only when apigateway.repo.createFromLocalRepo is true |
* (i.e all assets from the Source API Gateway) |
The List of assets that needs to be exported for building the assests from the Source API Gateway and build the repository. This is explained in detail in the below section. |
| 7 | apigateway.is.url | Optional, Mandated only when apigateway.repo.createFromLocalRepo is true | The URL for the IS where the API Gateway is hosted. | |
| 8 | apigateway.is.username | Optional, Mandated only when apigateway.repo.createFromLocalRepo is true | NA | The username to connect to the target. |
| 9 | apigateway.is.password | Optional, Mandated only when apigateway.repo.createFromLocalRepo is true | NA | The password to connect to the target. |
| 10 | apigateway.ssl.keystore.file | Optional | NA | Keystore to use when communication between Source API Gateway and ABE is over SSL. |
| 11 | apigateway.ssl.keystore.type | Optional | JKS | JKS Type of the keystore , when communication between Source API Gateway and ABE is over SSL. |
| 12 | apigateway.ssl.keystore.password | Optional | NA | Password for the keystore mentioned using 'apigateway.ssl.keystore.file' |
| 13 | apigateway.ssl.keyalias Optional | Optional | NA | Key alias from the keystore |
| 14 | apigateway.ssl.keyalias.password | Optional | NA | Password of the Key alias |
| 15 | apigateway.logging.level | Optional | off | Logging level for the ABE script |
Run the below command to create a repository for API Gateway assets.
bin> build.bat -Dbuild.output.dir=C:\apigatewayRepoForDeployer\ -Dapigateway.is.url=http://localhost:5555 -Dapigateway.is.username=Administrator -Dapigateway.is.password=manage
This will create a repository under C:\apigatewayRepoForDeployer\APIGateway which contains APIGatewayAssets.zip and the APIGatewayAssets.acdl files with all the assets that are present in the source API Gateway.
Create repository for only a subset of assets from the source API Gateway
It is possible to create only a subset of assets from the Source API Gateway and this can be done by using the apigateway.assets.file argument in the command line. The value of this parameter is a JSON file that will contain details of the assets that need to be exported from the source API Gateway.
bin> build.bat -Dbuild.output.dir=C:\apigatewayRepoForDeployer\ -Dapigateway.is.url=http://localhost:5555 -Dapigateway.is.username=Administrator -Dapigateway.is.password=manage -Dapigateway.assets.file=C:\assets.json
The content of the JSON file can be modified that will provide the flexibility for the users to select assets from the source API Gateway. The below table details few samples.
| # | Sample use case | JSON file content | Comments |
|---|---|---|---|
| 1 | Export all API Gateway assets | {} | It is equivalent to not passing this parameter as by default all assets will be pulled from the source gateway. |
| 2 | Export all APIs with consumed applications | { "types": [ "api" ] } |
This use case can be extended for all types of assets. The 'types' filed can take value
|
| 3 | Export all APIs without consumed applications | { "types": [ "api" ], "includeOptions": { "includeApplications": false } } |
|
| 4 | Export only APIs with name contains "API Gateway" | { "types": [ "api" ], "scope": [ { "attributeName": "apiName", "keyword": ".*API Gateway.*" } ], "includeOptions": { "includeApplications": false } } |
This example describes how we can search for assets based on keywords . We can also use Regex on the keyword field. |
| 5 | Export API with name starts with API Gateway and description contains search | { "types": [ "api" ], "scope": [ { "attributeName": "apiName", "keyword": "API Gateway.*" }, { "attributeName": "apiDescription", "keyword": ".*search.*" } ], "condition": "and", "includeOptions": { "includeApplications": false } } |
This example describes how we can search for assets based on multiple conditions. |
| 6 | Export all Policies excluding the system policies | { "types": [ "policy" ], "scope": [ { "attributeName": "policyScope", "keyword": "GLOBAL" } ] } |
This example is for users who want only the Global policies and not the system created ones. |
| 7 | Export all application without including associated APIs | { "types": [ "application" ], "includeOptions": { "includeApis": false } } |
This example is for users who want only the applications and not their associated APIs. |
| 8 | Export all packages along with subscription but with out the registered Applications for the associated APIs. | { "types": [ "package" ], "includeOptions": { "includeApis": false, "includeApplications": false, "includeSubscriptions": true } } |
Step 2 : Promoting the built assets using webmethods Deployer
Once the assets are built, the next step is to deploy the binary .zip file to the target API Gateways. This is done using webmethods Deployer. webMethods Deployer is a tool we can use to deploy the user created assets that reside on source API Gateway to target API Gateway. For example, you might want to deploy assets you have developed on servers in a development environment (the source) to servers in a test or production environment (the target). In such case, API Gateway uses the Repository based development in Deployer to build the assets from the development environment or from a Version Control System (VCS) and store them as a binary format in the flat file repository. This repository is built as explained in Step 1 and used in Deployer for deployment.
To start in Deployer we should need to define the Target API Gateways in Deployer. The below snapshot explains how this can be done.
After the target API Gateways are created , deployment would done in three steps Define, Map and Deploy.
Define: Defining the Repository
The first step is to define the repository containing the API Gateway asset binary that was created using ABE. Navigate to Deployer→Repository→ Add Repository and provide the location of the repository.
Define: Defining the project
As a part of this step we define the project for Deployment.
- Create a project from Deployer→Projects→Create project
- Select the type of the project as Repository based.
- Select the repository that we created from step 2.1.1 into this project.
We would need to create a Deployment set i.e the list of assets that needs to be deployed. WmDeployer will parse the repository and list all the assets that are available in this repository as shown below. We could select the assets that need to be deployed from the list.
Once we have selected the required assets and saved, the selected assets will be added to the Deployment set.
Map: Map the Target servers
In this step we map the target API Gateway servers to which the assets would be deployed. We would need to create a deployment map and then add the Target servers or a Target group to this map project.
Map: Variable substitutions
When we have an asset to be deployed to different stages, some variables of the asset might have different values for each stage. This is achieved using the Aliases in API Gateway.. In this case when an alias has different values specific to the stages, the Deployer allows us to input different values for different target servers during the Map step. Click on Configure to configure different values for different API Gateway aliases.
The mapping between the Stage and the Alias is done as shown below.
Deploy
The final step is to deploy the assets to the target API Gateway servers. We would need to create a deployment candidate and then Click on the Deploy image to deploy the assets.
With these three steps process of Define, Map and Deploy we have moved the assets from the source API Gateway to the target API Gateway. Apart from this typical usage, we also can simulate the deployment(which is a no op) and roll back to previous state by using the Simulate and Rollback options in the deployment candidate.
DevOps and CI/CD use case of API Gateway assets
In an effort towards modernization of the APIs and the organizations moving towards Agile from traditional waterfall, DevOps is becoming essential in making sure that the development and operations teams work towards a common goal. Achieving continuous integration and delivery is a key factor for an organization's success. The same can be achieved for API Gateway assets with the below two steps.
- Automated provisioning i.e. Automatic promotions of APIs, policies and other API Gateway assets from one stage to other
- Integration of the API assets with a source code management system like SVN or Github
There are two ways using which we can achieve DevOps use case of API Gateway assets.
- Using Import and Export APIs for CI/CD
- Using webmethods Deployer and ABE for CI/CD
Let's discuss them in detail in the below sections.
Using Import and Export APIs for CI/CD
One way of achieving Continuous integration of API Gateway assets is by using the Import and Export feature in API Gateway. API Gateway exposes APIs for Import and Export of APIs and other assets that can be used for CI/CD. The Export API on invocation on the source API Gateway would return an archive of the API Gateway assets. This archive could be stored in a repository like Nexus or Jfrog and then can be used as an input to the Import API of the target API Gateway servers.
We could have an Jenkins job configured to run this in an periodic fashion. The job would run command to invoke the Export API on the source API Gateway server to export the assets and store in a repository and another command to invoke the Import API on the target API Gateway server to import the assets stored in the repository.
Using webmethods Deployer and ABE for CI/CD
Consider an organization which develops APIs works in the following way for achieving continuous delivery.
- Developers concurrently and continuously work on their APIs in their development environments and submit their changes to a source code management system
- These changes are merged and deployed to a QA environment for further testing of the APIs
- An automation tool like Jenkins could read the API data from the VCS and with the help of the ABE and Webmethods Deployer, pushes this built data to the QA environment for testing
The following diagram explains the flow.
Flat file representation of API Gateway assets
In a DevOps managed deployment style, merging and comparing changes would be easy if the API Gateway assets are represented in a flat file form in the source code management system.
The flat file representation of API Gateway assets contains two main sections.
- The JSON object that contains the definition and object model of the asset
- The dependent objects of an asset
The dependent object can also contain its own dependent objects. For example an API object contains policy object as its dependent object and policy object contains Policy action objects as its dependent objects.
Creating flat file representation of API Gateway Assets using ABE
For enabling DevOps, ABE scripts for API Gateway provides more flexibility to
- Create a local representation of a source API Gateway
- Create a repository from this local representation
A local repository is built by passing the parameter apigateway.buildLocalRepoOnly=true to ABE build command with the location of the local repository. A sample usage is given below.
bin> build.bat -Dbuild.output.dir=C:\myVCSLocation\ -Dapigateway.buildLocalRepoOnly=true
Once a local repository is ready this can be used to build the repository needed for deployment in webmethods Deployer. This is done by passing apigateway.repo.createFromLocalRepo=true during the build of the repository. A sample usage is given below.
bin> build.bat -Dbuild.output.dir=C:\apigatewayRepoForDeployer\ -Dapigateway.repo.createFromLocalRepo=true -Dapigateway.repo.localRepo.path=C:\myVCSLocation
Deployer scripts for automation
Deployer provides scripts that helps in the automation of Define, Map and Deploy steps. These scripts are present in Deplyoer.bat|sh file in the installation path of the WmDeployer pacakge and provides almost all functionalities offered by Deployer. Please refer the Deployer guide for more details of this Project automation tool.
Limitations
There are some limitations with API Gateway assets for staging and promotion.
The following API Gateway assets cannot be staged using webmethods Deployer.
- Approval Configurations
- Access Profiles (but supported from 10.4)
The following API Gateway assets cannot be staged using API Gateway. In fact they have to be staged as Integration Server assets.
- Users
- Groups
- Keystores
- Truststores
- Outbound Proxy
- URL Aliases (but supported from 10.4)
Refer the Deployer documentation for more details on staging IS based assets.The following points are to be noted.
- APIs, Global Policies and Packages will be maintained in the same state in the target servers as they were when built using ABE.
- If the same API exists in active state, it cannot be redeployed using webMethods Deployer.
- Administrator settings and Assertions cannot be deleted using the deletion sets from Deployer.
- While deploying partial composites, the hard dependencies that are required, that is, Policy actions for Policies are added automatically. This behavior can not be overridden as of now.
References
- For a detailed understanding of ABE and webmethods Deployer and their installation please refer the user guide at https://documentation.softwareag.com/webmethods/wmsuites/wmsuite10-1/Deployer/10-1_Deployer_Users_Guide.pdf
- For creating a jenkins job with github as the VCS please refer https://www.packtpub.com/mapt/book/application_development/9781783553471/2/ch02lvl1sec23/creating-a-new-build-job-in-jenkins-with-git
- For configuring a github webhook with jenkins please refer https://thepracticalsysadmin.com/setting-up-a-github-webhook-in-jenkins/
Downloadable artifacts
The below videos provide a detailed demonstration of Staging, Promotion and DevOps (CI/CD) use case of API Gateway assets. Please find the video files in the attachments.
Learn more
- If you want to learn more on this topic and watch demos please register the course in the Software AG corporate university at https://learning.softwareag.com/course/view.php?id=1839
