Author: Raj, Gokul (gora@softwareag.com)
Supported Versions: 10.5 and above
Overview of the tutorial
API Gateway provides different policies at different stages of the API flow which is sufficient enough for security, transformation, validation, error processing and monitoring. In addition to the existing policies, API Gateway now supports Custom Extension policy using which the customer can invoke their custom logic from API Gateway. This tutorial describes the details of different custom extension types that can be added to an API.
In this tutorial we will go through the various Custom Extension types and their usage in detail.
Required knowledge
The tutorial assumes that the reader has,
- a basic understanding of API Gateway and its policy enforcement
- a good knowledge on APIs and their definitions
Why?
API Gateway provides a set of policies which are more than sufficient to develop an API which meets most of the customer requirements. But sometimes the customer may want to plug in their custom business logic into API Gateway policy enforcement to accomplish their tasks which might not be handled by any of the policies. This requirement drives API Gateway to provide Custom Extension support to allow the customer to implement a custom logic and make a call out to it by configuring a Custom Extension policy in the API Gateway policy enforcement.
The use cases are endless. Let's see some sample usage scenarios to understand it clearly.
- As you already know, API Gateway already comes with a lot of security policies to be applied for an API. But in some scenarios that the customer may feel that the supported out-of-the-box policies are not sufficient for their API in that context and want to impose their own security policy on that API which they have hosted on an AWS Lambda function. In such cases they can use the Custom Extension policy in the Identify & Access stage of the API policy execution flow and configure it to invoke the AWS Lambda function which hosts the customer's legacy security policy to provide a customized security protection to their API.
- API Gateway provides Transformation policies in Request and Response processing stages to transform an XML payload using XSLT transformation framework which is a standardized one. But API Gateway doesn't have a transformation policy to transform a JSON payload because there are lot of proprietary approaches and solutions to accomplish this but a standard method is not yet available and hence this feature is not included in the product. In such case, the customer can provide their own custom logic to transform the JSON payload by using any of the third party solutions available, expose it as a REST API and configure it using External Endpoint in the Custom Extension policy of the API.
- Another use case would be, say for example, the customer wants to post a part of the request or response detail to a JMS queue and later want to process it to accomplish multiple tasks like a customized transaction logging, triggering an action based on the detail, etc. This can be achieved by using Messaging type in the Custom Extension policy.
Prerequisite steps
- Install API Gateway version 10.5 or above if the reader uses on-premise installation (Note: Custom Extension policy is supported in cloud as well)
Details
In this section let us see the discuss the details of different custom extension types provided by API Gateway. The product supports 4 custom extension types as of 10.5.
- External endpoint
- webMethods IS service
- AWS Lambda
- Messaging
Some of the key points to note about the Custom Extension policy are,
- Supported for SOAP, REST and ODATA APIs
- Supported in all stages except Transport & Traffic Monitoring
- No restriction on the number of Custom Extension policies in a particular stage
In the next sections we will take External endpoint and explore it with a sample use case. Other types are explained with their configuration in detail.
Type 1: External endpoint
In this type the customer would have their custom business logic in REST APIs. The REST endpoint can be configured in Custom Extension policy and the same would be invoked on API invocation. Only REST APIs are supported in external endpoint. For our understanding let's take the JSON transformation use case which we outlined above and create an API to invoke the custom logic HTTP service using Custom Extension policy. The steps are given below.
Step 1: Run the sample service
Run the Java spring application for the external endpoint.
$> java -jar jsonservice.jar
This application exposes a REST endpoint to convert the JSON payload. For our use case it will add the property key value "id":"7176" to the payload and converts the "photoUrls" object from string to string array.
POST http://localhost:8080/services/jsonTransformation
Step 2: Create the API
Login to API Gateway and create a Swagger Petstore API from the swagger definition URL http://petstore.swagger.io/v2/swagger.json. Edit the API, go to Policies → Request Processing. Add Custom Extension policy. The Custom Extension policy can be added in all stages except Transport & Traffic Monitoring and there is no restriction on the number of Custom Extension policies in a particular stage. The API is available as SwaggerPetstore_API in the attachment section.
Step 3: Configure External endpoint
Select External endpoint in the Type. Provide the sample service URL value http://localhost:8080/services/jsonTransformation in the Endpoint URI and select Method as POST. We can also use Endpoint and Simple alias in Endpoint URI.
This endpoint configuration has different properties depending on the Custom Extension type. Apart from the Endpoint URI of the external REST API there are few other configurations available. They can be configured depending on the requirements.
SSL Configuration
We can configure Keystore and Truststore details, if the external endpoint is exposed over HTTPS protocol. For this, first we need to add the keystore and truststore details in API Gateway administration section. Moreover we can control the Connection and Read timeout values for the endpoint.
Path Parameters
If the REST endpoint has path parameters, we can configure them in this section. The path parameter value can be given using variable framework which is defined later in this tutorial.
Step 4: Configure the request payload to the external REST API
In this step we will configure the payload that would be sent to the external REST API. Under Request Processing, expand the Payload section. Provide the payload ${request.payload} in the Payload section. This will send the whole payload from the main request to the external endpoint. We can either send the entire payload or we can extract the values from the request and response payload and construct a new payload for the external endpoint using variable framework. Variable framework is explained in detail later in this step.
Payload
We can provide the payload in two ways, Inline Request and Load from Schema. For a better view click the expand icon on the right top of the payload section.
Inline Request
Here the actual payload can be given that would be sent to the external endpoint.
Load from Schema
Here we can generate the payload from a json or xml schema. Browse and select the schema file and click save. It will generate the payload with random values.
Variable types and aliases
In this section we can discuss how the values such as headers, query parameters, etc, can be accessed. The below table shows the variable types and their possible values.
|
Object/Variable |
Type Possible values |
|---|---|
| paramStage |
|
| paramType |
|
| queryType |
|
${paramStage.paramType}
You can use this syntax to access the following string variables: path, statusCode, statusMessage, httpMethod. Examples: ${request.path}, ${response.statusCode}
${paramStage.paramType.paramName}
You can use this syntax to access map types, such as query, headers, and path. Example: ${request.query.var1}, ${response.header.Content-Type}, ${request.path.name}.
${paramStage.paramType.queryType[queryValue]}
This syntax can be used to query a paramType. Examples:
- ${request.payload.xpath[//ns:emp/ns:empName]} - Where "//ns:emp/ns:empName" is the XPath to be applied on the payload if contentType is "application/xml", "text/xml", or "text/html".
- ${response.payload.jsonPath[$.cardDetails.number]} - Where "$.cardDetails.number" is the jsonPath to be applied on payload if contentType is "application/json" or "application/json/badgerfish".
- ${request.payload.regex[[0-9]+]} - Where "[0-9]+" is the regular expression to be applied on the payload if contentType is "text/plain".
${paramStage.paramType.queryType[queryValue]}
You can use this syntax to access response of the previous custom extension. For example, you can use the following syntax to access the response payload of the previous custom extension: ${response[customExtension].payload.jsonPath[$.echo]}.
| paramStage |
paramType |
extra parameters |
|---|---|---|
request |
headers | HEADER_NAME |
| query (applicable only for REST) | QUERY_NAME | |
|
path (applicable only for REST)
|
PATH_PARAMETER | |
| regex[REGEX_EXPRESSION] | ||
|
PATH STRING |
||
| httpMethod (applicable only for REST) | - | |
|
payload |
xpath[EXPRESSION] | |
| jsonPath[EXPRESSION] | ||
| regex[EXPRESSION] | ||
|
authorization |
clientId |
|
|
issuer |
||
|
userName |
||
|
authHeader |
||
|
incomingToken |
||
|
audience (This was introduced to hold the audience value calculated after the authorization process. In case of JWT/OpenID/OAuth2 this would be the "aud" claim. For SAML or other type or authentication it would be the audience URI. But because of time constraint, it was not populated. In future this would hold the audience of the token.) |
||
|
claims.CLAIM_NAME |
||
response |
header | HEADER_NAME |
|
statusCode |
- | |
|
statusMessage |
- | |
|
payload |
jsonPath[EXPRESSION] | |
| regex[EXPRESSION] | ||
|
xpath[EXPRESSION] |
Step 5: Configure headers & Query parameters to the external REST API
In this step we will configure the request headers that would be sent to the external REST API. Expand Headers section and select Use incoming headers. This will pass all the main incoming request headers to the external endpoint. If not selected, it will not pass any headers from the request to the external endpoint.
The customer can also configure custom headers that needs to be sent to the external endpoint in the below section. Header values can be given using variables. If same header is present in Incoming request and Custom Headers section, then the header in the Custom Headers section will take preference over the incoming header.
Query Parameters
If the endpoint has query parameters, then we can configure query parameters using this section. Query parameter values can be given using variables.
Step 6: Configure the response payload
Expand Response Processing, select Copy entire response. If selected, this will copy the entire response payload from the external REST endpoint and replace the existing request payload.
Transformation
Instead of copying the entire payload to the request flow the user can use Transformation section to manipulate the request payload with the values from external REST API response using variables and XPATH or JSON Path expressions. Let's see an example.
Response payload from Custom Extension external endpoint: {"petName":"doggie"}
Main request payload: {"id":"100","name":"pet"}
Updated payload after transformation: {"id":"100","name":"doggie"}
To transform the above payload we can use JSON path expression. Define the Variable as ${request.payload.jsonPath[$.name]} and the Value as ${response[customExtension].payload.jsonPath[$.petName]}. This will fetch the value doggie from the response of the external endpoint and replace the value pet in the existing payload.
Custom variable creation and scope
We can also create a custom variable in Transformation section. The created custom variable can be accessed in other Custom Extension policies in the policy execution flow. The custom variable should be accessed using ${variable name} syntax. For e.g $(PetName} for the variable created in the below screenshot.
Custom Extension Metadata
This section helps to configure the namespaces for an XML payload transformation. Let's see an example
- Variable: ${request.payload.xpath[//axis:name]}
- Value: ${response.payload.jsonPath[$.echo]}
For the above configured transformation, we need to provide the Namespace URI for Namespace Prefix axis as given below.
Step 7: Abort API execution in case of failure
Under Response Processing, select Abort API execution in case of failure. The detail is given below.
|
Abort API execution in case of failure |
Custom Extension failed
|
|---|---|
| Selected |
In case the Custom Extension policy returns an error status code, this will abort the main policy enforcement flow, stops further execution of policies and returns 500 Internal server error to the client. |
| Not selected | In case the Custom Extension policy returns an error status code, It will just logs the error and move on to the next policy execution in policy enforcement flow. |
Step 8: Custom Extension policy in action
Invoke the API using a REST client.
URL: http://localhost:5555/gateway/Swagger%20Petstore/1.0.3/pet
Method: POST
Username and Password: admin
Payload:
{
"name": "custom extension pet",
"status": "available",
"photoUrls": "http://petstore.swagger.io/pet.jpg"
}
From the response you can see that the external endpoint was invoked and the response is fed into the main request flow of the API invocation.
Type 2: webMethods IS service
The webmethods IS service Custom Extension type does exactly the same as Invoke webmethods IS policy, that is, using this type we can invoke a webMethods IS service. Unlike other Custom Extension types we can't change either the request or response payloads in this type. For more details on Invoke webmethods IS policy please refer Invoke webMethods IS policy in API Gateway 10.2. Lets see this in action.
Step 1: Run the sample service
A sample echo service is attached in this tutorial. Install the IS package in IS admin page.
Download the custom extension service package CustomExtension_Service.zip and place it in the location <InstallationLocation>/IntegrationServer/instances/{instanceName}/replicate/inbound. Open IS admin page http://localhost:5555, go to Packages → Management → Install Inbound Releases and install the custom extension service package. Once installed, return to services list page. There you can see the package installed and loaded successfully.
Step 2: Configure the Custom Extension policy
Create an API and add the Custom Extension policy. Select webMethods IS service in the Type. Click on Add webMethods is service and provide the IS service and the user used to run the service. Click Add to add the configured IS service.
webMethods IS service alias (Optional)
Instead of configuring IS service details every time, we can create an alias for webMethods IS service and use it here. Refer Invoke webMethods IS policy in API Gateway 10.2 on how to create a webMethods IS service alias. Choose the created alias from the drop down.
Step 3: Custom Extension policy in action
Save and activate the API. Invoke the API with a REST client.
Type 3: AWS Lambda
AWS Lambda is a compute service used to run code without provisioning or managing server. You can write your application code in languages supported by AWS Lambda, and run within the AWS Lambda standard runtime environment and resources provided by Lambda. As mentioned earlier, customer can have their custom logic running in AWS Lambda function. API Gateway provides support to invoke the Lambda functions from the API execution flow through Custom Extension policy.
AWS Lambda alias
To invoke a Lambda function, we need to create a AWS account configuration in the API Gateway Administration section with the Access key ID, Secret access key and Region. This can be created by navigating to Administration → External accounts → AWS configuration. Configure the AWS account details here and use it as an alias in the Custom Extension policy. Customer can configure multiple AWS accounts.
Policy configuration
The configuration for AWS Lambda Custom Extension type is explained below in detail.
Function Name
This is the AWS Lambda function name that you want to invoke during the API execution flow.
Invocation Type
Two types of invocation are supported - RequestResponse and Event. RequestResponse is synchronous and Event is asynchronous.
AWS Alias
AWS configuration for connecting to the AWS account which hosts the Lambda function.
Client Configuration
These are the configurations for the AWS Lambda client in API Gateway which are useful when making a connection to the AWS Lambda function. Select the client configuration from drop down or provide custom client configuration as AWS introduces.
For AWS Client Configuration please refer https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html and https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/ClientConfiguration.html.
Type 4: Messaging
This Custom Extension type helps to post a message to a JMS or AMQP provider queue in the API execution flow. Please refer AMQP 1.0 support in API Gateway & SOAP over JMS in API Gateway for creating the JMS or AMQP related configurations to communicate with a JMS or AMQP provider. The configuration for Messaging Custom Extension type is explained below in detail.
Connection Alias Name
Name of the connection. Connection Alias which we have created under Messaging administration configuration.
Destionation Name
Name of the Queue or Topic to post/route the request message.
Destination Type
Type of the destination, Either Queue or Topic.
ReplyTo Name
Name of the Queue or Topic to which the API Gateway look for the response message for the earlier posted request to the Destination Name.
ReplyToType
Type of the destination, Either Queue or Topic.
Time to Live
It sets the time to live of a message put onto a queue/topic, in milliseconds. A value of means live indefinitely.
Time to Wait
It sets the waiting time to the API Gateway for the response message.
Delivery Mode
The delivery modes, persistent or non persistent. The Persistent delivery mode, instructs the JMS provider to take extra care to ensure that a message is not lost in transit in case of a JMS provider failure. A message sent with this delivery mode is logged to stable storage when it is sent.
Limitations
- Custom Extension policy is not supported for Websocket API
Troubleshooting
|
#
|
Error/Exception
|
Message
|
Possible solution
|
|---|---|---|---|
| 1 |
{ "Exception": "API Gateway encountered an error. Error Message: API Gateway is unable to process incoming request. Request Details: Service - Swagger Petstore, Operation - ***********, Invocation Time:*********, Date:**********, Client IP - **********, User - Default and Application:null" } |
500 Internal server error |
In Response Processing, if we select the Abort API execution in case of failure, it will abort the API execution flow and returns an 500 Internal server error to the client.
Check whether the configured Custom Extension type is valid. |
References
- AWS Lambda Documentation
- https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html
- https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/index.html?com/amazonaws/ClientConfiguration.html
- https://docs.aws.amazon.com/lambda/latest/dg/welcome.html
Downloadable artifacts
Learn more
- For details on usage of Invoke webMethods IS policy in versions 10.2 and above, refer Invoke webMethods IS policy in API Gateway 10.2
- For details on AMQP, refer AMQP 1.0 support in API Gateway
