Custom Extension policy in API Gateway

Author: Raj, Gokul (
Supported Versions: 10.5 and above

Overview of the tutorial

API Gateway provides different policies at different stages of the API flow which is sufficient enough for security, transformation, validation, error processing and monitoring. In addition to the existing policies, API Gateway now supports Custom Extension policy using which the customer can invoke their custom logic from API Gateway. This tutorial describes the details of different custom extension types that can be added to an API. 

In this tutorial we will go through the various Custom Extension types and their usage in detail.

Required knowledge

The tutorial assumes that the reader has,

  • a basic understanding of API Gateway and its policy enforcement
  • a good knowledge on APIs and their definitions


API Gateway provides a set of policies which are more than sufficient to develop an API which meets most of the customer requirements. But sometimes the customer may want to plug in their custom business logic into API Gateway policy enforcement to accomplish their tasks which might not be handled by any of the policies. This requirement drives API Gateway to provide Custom Extension support to allow the customer to implement a custom logic and make a call out to it by configuring a Custom Extension policy in the API Gateway policy enforcement.

The use cases are endless. Let's see some sample usage scenarios to understand it clearly.

  • As you already know, API Gateway already comes with a lot of security policies to be applied for an API. But in some scenarios that the customer may feel that the supported out-of-the-box policies are not sufficient for their API in that context and want to impose their own security policy on that API which they have hosted on an AWS Lambda function. In such cases they can use the Custom Extension policy in the Identify & Access stage of the API policy execution flow and configure it to invoke the AWS Lambda function which hosts the customer's legacy security policy to provide a customized security protection to their API.
  • API Gateway provides Transformation policies in Request and Response processing stages to transform an XML payload using XSLT transformation framework which is a standardized one. But API Gateway doesn't have a transformation policy to transform a JSON payload because there are lot of proprietary approaches and solutions to accomplish this but a standard method is not yet available and hence this feature is not included in the product. In such case, the customer can provide their own custom logic to transform the JSON payload by using any of the third party solutions available, expose it as a REST API and configure it using External Endpoint in the Custom Extension policy of the API.
  • Another use case would be, say for example, the customer wants to post a part of the request or response detail to a JMS queue and later want to process it to accomplish multiple tasks like a customized transaction logging, triggering an action based on the detail, etc. This can be achieved by using Messaging type in the Custom Extension policy.

Prerequisite steps

  • Install API Gateway version 10.5 or above if the reader uses on-premise installation (Note: Custom Extension policy is supported in cloud as well)


In this section let us see the discuss the details of different custom extension types provided by API Gateway. The product supports 4 custom extension types as of 10.5.

  1. External endpoint
  2. webMethods IS service
  3. AWS Lambda
  4. Messaging

Some of the key points to note about the Custom Extension policy are,

  • Supported for SOAP, REST and ODATA APIs
  • Supported in all stages except Transport & Traffic Monitoring
  • No restriction on the number of Custom Extension policies in a particular stage

In the next sections we will take External endpoint and explore it with a sample use case. Other types are explained with their configuration in detail.

Type 1: External endpoint

In this type the customer would have their custom business logic in REST APIs. The REST endpoint can be configured in Custom Extension policy and the same would be invoked on API invocation. Only REST APIs are supported in external endpoint. For our understanding let's take the JSON transformation use case which we outlined above and create an API to invoke the custom logic HTTP service using Custom Extension policy. The steps are given below.

Step 1: Run the sample service

Run the Java spring application for the external endpoint.

$> java -jar jsonservice.jar

This application exposes a REST endpoint to convert the JSON payload. For our use case it will add the property key value "id":"7176" to the payload and converts the "photoUrls" object from string to string array.

POST http://localhost:8080/services/jsonTransformation

Step 2: Create the API

Login to API Gateway and create a Swagger Petstore API from the swagger definition URL Edit the API, go to Policies → Request Processing. Add Custom Extension policy. The Custom Extension policy can be added in all stages except Transport & Traffic Monitoring and there is no restriction on the number of Custom Extension policies in a particular stage. The API is available as SwaggerPetstore_API in the attachment section.

Step 3: Configure External endpoint

Select External endpoint in the Type. Provide the sample service URL value http://localhost:8080/services/jsonTransformation in the Endpoint URI and select Method as POST. We can also use Endpoint and Simple alias in Endpoint URI.

This endpoint configuration has different properties depending on the Custom Extension type. Apart from the Endpoint URI of the external REST API there are few other configurations available. They can be configured depending on the requirements.

SSL Configuration

We can configure Keystore and Truststore details, if the external endpoint is exposed over HTTPS protocol. For this, first we need to add the keystore and truststore details in API Gateway administration section. Moreover we can control the Connection and Read timeout values for the endpoint.

Path Parameters

If the REST endpoint has path parameters, we can configure them in this section. The path parameter value can be given using variable framework which is defined later in this tutorial.

Step 4: Configure the request payload to the external REST API

In this step we will configure the payload that would be sent to the external REST API. Under Request Processing, expand the Payload section. Provide the payload ${request.payload} in the Payload section. This will send the whole payload from the main request to the external endpoint. We can either send the entire payload or we can extract the values from the request and response payload and construct a new payload for the external endpoint using variable framework. Variable framework is explained in detail later in this step.


We can provide the payload in two ways, Inline Request and Load from Schema. For a better view click the expand icon on the right top of the payload section.

Inline Request

Here the actual payload can be given that would be sent to the external endpoint.

Load from Schema

Here we can generate the payload from a json or xml schema. Browse and select the schema file and click save. It will generate the payload with random values.

Variable types and aliases

In this section we can discuss how the values such as headers, query parameters, etc, can be accessed. The below table shows the variable types and their possible values.


Type Possible values

  • request
  • response
  • payload or body
  • headers
  • query
  • path
  • httpMethod
  • statusCode
  • statusMessage
  • xpath
  • jsonPath
  • regex


You can use this syntax to access the following string variables: path, statusCode, statusMessage, httpMethod. Examples: ${request.path}, ${response.statusCode}


You can use this syntax to access map types, such as query, headers, and path. Example: ${request.query.var1}, ${response.header.Content-Type}, ${}.


This syntax can be used to query a paramType. Examples:

  • ${request.payload.xpath[//ns:emp/ns:empName]}  - Where "//ns:emp/ns:empName" is the XPath to be applied on the payload if contentType is "application/xml", "text/xml", or "text/html".
  • ${response.payload.jsonPath[$.cardDetails.number]} - Where "$.cardDetails.number" is the jsonPath to be applied on payload if contentType is "application/json" or "application/json/badgerfish".
  • ${request.payload.regex[[0-9]+]} - Where "[0-9]+" is the regular expression to be applied on the payload if contentType is "text/plain".
While xpath and jsonPath are applicable only to payload, regEx can be used with both payload and path


You can use this syntax to access response of the previous custom extension. For example, you can use the following syntax to access the response payload of the previous custom extension: ${response[customExtension].payload.jsonPath[$.echo]}.



extra parameters


headers  HEADER_NAME
query (applicable only for REST) QUERY_NAME


path (applicable only for REST)




httpMethod (applicable only for REST) -













audience (This was introduced to hold the audience value calculated after the authorization process. In case of JWT/OpenID/OAuth2 this would be the "aud" claim. For SAML or other type or authentication it would be the audience URI. But because of time constraint, it was not populated. In future this would hold the audience of the token.)












Step 5: Configure headers & Query parameters to the external REST API

In this step we will configure the request headers that would be sent to the external REST API. Expand Headers section and select Use incoming headers. This will pass all the main incoming request headers to the external endpoint. If not selected, it will not pass any headers from the request to the external endpoint.

The customer can also configure custom headers that needs to be sent to the external endpoint in the below section. Header values can be given using variables. If same header is present in Incoming request and Custom Headers section, then the header in the Custom Headers section will take preference over the incoming header.

Query Parameters

If the endpoint has query parameters, then we can configure query parameters using this section. Query parameter values can be given using variables.

Step 6: Configure the response payload

Expand Response Processing, select Copy entire response. If selected, this will copy the entire response payload from the external REST endpoint and replace the existing request payload.


Instead of copying the entire payload to the request flow the user can use Transformation section to manipulate the request payload with the values from external REST API response using variables and XPATH or JSON Path expressions. Let's see an example.

Response payload from Custom Extension external endpoint: {"petName":"doggie"}

Main request payload: {"id":"100","name":"pet"}

Updated payload after transformation: {"id":"100","name":"doggie"}

To transform the above payload we can use JSON path expression. Define the Variable as ${request.payload.jsonPath[$.name]} and the Value as ${response[customExtension].payload.jsonPath[$.petName]}. This will fetch the value doggie from the response of the external endpoint and replace the value pet in the existing payload.

Custom variable creation and scope

We can also create a custom variable in Transformation section. The created custom variable can be accessed in other Custom Extension policies in the policy execution flow. The custom variable should be accessed using ${variable name} syntax. For e.g $(PetName} for the variable created in the below screenshot. 

Custom Extension Metadata

This section helps to configure the namespaces for an XML payload transformation. Let's see an example

  • Variable: ${request.payload.xpath[//axis:name]}
  • Value: ${response.payload.jsonPath[$.echo]}

For the above configured transformation, we need to provide the Namespace URI for Namespace Prefix axis as given below.

Step 7: Abort API execution in case of failure

Under Response Processing, select Abort API execution in case of failure. The detail is given below.

Abort API execution in case of failure

Custom Extension failed

In case the Custom Extension policy returns an error status code, this will abort the main policy enforcement flow, stops further execution of policies and returns 500 Internal server error to the client.

Not selected In case the Custom Extension policy returns an error status code, It will just logs the error and move on to the next policy execution in policy enforcement flow.

Step 8: Custom Extension policy in action

Invoke the API using a REST client.

URL: http://localhost:5555/gateway/Swagger%20Petstore/1.0.3/pet

Method: POST

Username and Password: admin


  "name": "custom extension pet",
  "status": "available",
  "photoUrls": ""

From the response you can see that the external endpoint was invoked and the response is fed into the main request flow of the API invocation.

Type 2: webMethods IS service

The webmethods IS service Custom Extension type does exactly the same as Invoke webmethods IS policy, that is, using this type we can invoke a webMethods IS service. Unlike other Custom Extension types we can't change either the request or response payloads in this type. For more details on Invoke webmethods IS policy please refer Invoke webMethods IS policy in API Gateway 10.2. Lets see this in action.

Step 1: Run the sample service

A sample echo service is attached in this tutorial. Install the IS package in IS admin page.

Download the custom extension service package and place it in the location <InstallationLocation>/IntegrationServer/instances/{instanceName}/replicate/inbound. Open IS admin page http://localhost:5555, go to Packages → Management → Install Inbound Releases and install the custom extension service package. Once installed, return to services list page. There you can see the package installed and loaded successfully.

Step 2: Configure the Custom Extension policy

Create an API and add the Custom Extension policy. Select webMethods IS service in the Type. Click on Add webMethods is service and provide the IS service and the user used to run the service. Click Add to add the configured IS service.

webMethods IS service alias (Optional)

Instead of configuring IS service details every time, we can create an alias for webMethods IS service and use it here. Refer Invoke webMethods IS policy in API Gateway 10.2 on how to create a webMethods IS service alias. Choose the created alias from the drop down.

Step 3: Custom Extension policy in action

Save and activate the API. Invoke the API with a REST client.

Type 3: AWS Lambda

AWS Lambda is a compute service used to run code without provisioning or managing server. You can write your application code in languages supported by AWS Lambda, and run within the AWS Lambda standard runtime environment and resources provided by Lambda. As mentioned earlier, customer can have their custom logic running in AWS Lambda function. API Gateway provides support to invoke the Lambda functions from the API execution flow through Custom Extension policy.

AWS Lambda alias

To invoke a Lambda function, we need to create a AWS account configuration in the API Gateway Administration section with the Access key ID, Secret access key and Region. This can be created by navigating to Administration → External accounts → AWS configuration. Configure the AWS account details here and use it as an alias in the Custom Extension policy. Customer can configure multiple AWS accounts.

Policy configuration

The configuration for AWS Lambda Custom Extension type is explained below in detail.

Function Name

This is the AWS Lambda function name that you want to invoke during the API execution flow.

Invocation Type

Two types of invocation are supported - RequestResponse and Event. RequestResponse is synchronous and Event is asynchronous.

AWS Alias

AWS configuration for connecting to the AWS account which hosts the Lambda function.

Client Configuration

These are the configurations for the AWS Lambda client in API Gateway which are useful when making a connection to the AWS Lambda function. Select the client configuration from drop down or provide custom client configuration as AWS introduces.

For AWS Client Configuration please refer and

For all other steps like Request Processing, Response Processing and Custom Extension Metadata please refer External Endpoint section

Type 4: Messaging

This Custom Extension type helps to post a message to a JMS or AMQP provider queue in the API execution flow. Please refer AMQP 1.0 support in API Gateway & SOAP over JMS in API Gateway for creating the JMS or AMQP related configurations to communicate with a JMS or AMQP provider.  The configuration for Messaging Custom Extension type is explained below in detail.

Connection Alias Name

Name of the connection. Connection Alias which we have created under Messaging administration configuration.

Destionation Name

Name of the Queue or Topic to post/route the request message.

Destination Type

Type of the destination, Either Queue or Topic.

ReplyTo Name

Name of the Queue or Topic to which the API Gateway look for the response message for the earlier posted request to the Destination Name.


Type of the destination, Either Queue or Topic.

Time to Live

It sets the time to live of a message put onto a queue/topic, in milliseconds. A value of  means live indefinitely.

Time to Wait

It sets the waiting time to the API Gateway for the response message.

Delivery Mode

The delivery modes, persistent or non persistent. The Persistent delivery mode, instructs the JMS provider to take extra care to ensure that a message is not lost in transit in case of a JMS provider failure. A message sent with this delivery mode is logged to stable storage when it is sent.

For all other steps like Request Processing, Response Processing and Custom Extension Metadata please refer External Endpoint section


  • Custom Extension policy is not supported for Websocket API


Possible solution


"Exception": "API Gateway encountered an error. Error Message:  API Gateway is unable to process incoming request. Request Details:  Service - Swagger Petstore, Operation - ***********, Invocation Time:*********, Date:**********,  Client IP - **********, User - Default and Application:null"


500 Internal server error

In Response Processing, if we select the Abort API execution in case of failure, it will abort the API execution flow and returns an 500 Internal server error to the client.


Check whether the configured Custom Extension type is valid.


Downloadable artifacts

Learn more



Custom Extension.PNG (27.6 KB)












jsonservice.jar (20.4 MB)





Swagger (10.4 KB)





Hi Raj,

Thanks for the post, however the custom policy for Type 3 : AWS lambda is not working as per the steps mentioned in this page.

There are two issue I face while I do a setup for it

  1. Native Endpoint : This field is marked as mandatory and needs to be provided. However for lambda function there is no Native endpoint. As you might know it works on the arn of the lambda function.

  2. Routing : While using custom extension in the Routing to invoke lambda function and saving it, product automatically puts in the Straight through Policy. Due to which it appends the native URL as well in the endpoint (eg: /npe/${sys:resource_path} ) .

I have been trying set this up and this is still not working. Page needs to be more clear if possible even the SAG documentation is not clear with the steps.

I also tried to import the attached custom policy zip file. but it throws an error. Probably due to API Gateway version mismatch. I would be good know the product version with the page this setup has been tried.

Hi Raj,

We have a scenario where we are creating a pass-through service and in this pass-through, native service has it’s own authentication mechanism. As per the native API authentication, I have to pass signature (bas64 of the request payload) as custom header.

I would like to know weather is it possible or not… if yes I would like to know the procedure of generating this signature bye converting request payload base64.

Thank you.

Hi Atif,

You can write an IS Service to base64 encode the payload and pass it via a custom header to the native server. You can do this via Invoke webMethods IS policy (and I think, Invoke IS in custom extension policy would also help you to do the same).

For more details on usage of Invoke webMethods IS policy, refer this link. Invoke webMethods IS policy in API Gateway 10.2


Dear Vallab,

Thank you for the reply. I wonder if there’s any possibility that we can achieve the same through API gateway only? I mean is there any configuration there to convert the headers to base64.

Also in another scenario we have to sign the part of request payload with certificate.

Do we have something configurable in API gateway itself to achieve above scenarios without involving IS?


Atif Minhas