Apama connectivity plug-ins

Knowledge base
, ,
1

Rapidly build adapters to connect external systems

Kevin Palfreyman, VP R&D, Apama and Universal Messaging, Software AG
Chris Reed, Senior Architect, Apama, Software AG

Do more with Apama by building adapters to external systems, whether they are in the cloud, local or provided via third parties, and stream complex events to analyze, detect patterns and take action.
 

What is Apama?

Apama is Software AG's award winning Complex Event Processing (CEP) and streaming analytics product, which detects complex patterns of events and takes actions on those patterns, all in real time. This product is particularly applicable to the types of high-velocity data often seen in the industrial IoT, finance and fraud detection. Apama recently released Apama Community Edition, a free edition of Apama.
 

Existing connectivity options for Apama

The core CEP component of Apama is known as the "correlator". This standalone server process is purposely built to efficiently process event streams. Apama has a number of established connectivity options that provide mechanisms to integrate with external systems that may be the source of events supplied to Apama or the destination for events produced by Apama.

The Integration Adapter Framework (IAF)

The IAF is a separate process for Java® or C adapters that are composed of a "transport" and a "codec". The codec transforms between the external format and Apama events that are passed to and from the correlator.

Correlator client APIs

These APIs allow other processes to connect to a correlator in order to send and receive Apama events. It is available for C, C++, Java and Microsoft®.Net environments.

Correlator-integrated messaging for JMS

This connects as a client to a Java® Messaging Service (JMS) bus with an option for reliable messaging. This mechanism is also used to integrate with other Software AG Digital Business Platform components, such as webMethods Integration Server.

Native connection

Finally, the correlator is also able to connect natively as a client to the Software AG Universal Messaging (UM) bus, without JMS, to send and receive Apama events.

In each of the previous options, except JMS, the wire-encoded format of events entering the correlator is the Apama event format. These events are parsed as they enter the correlator before being filtered at the primary indexing structure (known as the hyper-tree) and forwarded to interested applications inside the correlator.
 

Introducing Apama "Connectivity Plug-ins"

Apama 9.10, released in April 2016, introduced a new option for integration with external systems via a feature called "connectivity plug-ins" which are very easy to use interfaces for building adapters to external systems. External systems may be cloud services accessible over RESTful HTTP, local hardware sensors on the same host as Apama, or any other service or application that may be accessed via a third-party library or may be streaming data into Apama.

These connectivity plug-ins are hosted in-process in the correlator which means that, compared with the IAF, you have a simpler DevOps experience with fewer processes to configure, deploy, manage, monitor, etc. You also have the potential for lower latency by removing an inter-process message transfer. Also, compared to correlator-integrated messaging for JMS (or native UM connectivity), connectivity plug-ins offer a greater degree of flexibility via multiple codecs. These plug-ins allow events to be sent and received between the plug-in and the correlator in which they are deployed in a simple "map" structure, without requiring a separate parsing step, which also helps increase overall performance.
 

Architecture of connectivity plug-ins

Connectivity plug-ins encourage separation of concerns, separating the transport from the format of the messages. For example, a JSONCodec can be used with a client HTTPTransport (both are included as source code samples in the product). Connectivity plug-ins can be written in Java or C++, or even mix Java and C++ plug-ins together to provide a useful adapter.

Connectivity plug-ins are used in "chains" that consist of a transport, any number of codecs and a host plug-in. The “host plug-in” is the connectivity chain's interface to the correlator internals. The chain is defined in a YAML file along with a configuration specifying where to load the plug-ins and a configuration for each plug-in in the chain.

If we consider an event sent from the Apama correlator out towards an external system, then messages are passed through the chain from host plug-in, on to one or more codecs, and then on to a transport. Each message consists of a payload object and metadata.

The "apama.eventMap" host plug-in translates Apama's internal event format into map payload objects, where the keys of the map are the event field names and the values are mapped from the EPL values--in Java, these will be String, Double, Long, Map, etc. objects. The metadata holds the event type used for the event, the channel the event is to be delivered on (allowing routing of different events into different parts of the application running inside the correlator), and can be used to hold adapter-specific data (such as HTTP headers).

Codecs may change the payload or metadata, for example:

  • Convert the payload into a serializable form such as string  or an array of bytes (the JSON Codec performs this job in the above example)
     
  • Re-arrange the contents of the payload and metadata (mapperCodec)
     
  • Identify the event type based on metadata or message data (classifierCodec)

Connectivity plug-ins are well suited to handling a variety of styles of integration such as:

  • Streaming data
  • Request/response which can be used to enrich incoming data
  • Invoking requests against a third-party library
  • One-way input data or actions
  • Periodically retrieving data from an external service
     

The weather sample

To illustrate connectivity plug-ins, let's look at the weather sample found in the Apama/samples/connectivity_plugin/application/weather folder in a product installation. The weather sample provides four plug-ins as samples to send HTTP requests to two external services. These requests go to freegeoip.net to perform a GeoIP lookup and to openweathermap.org to find the current weather where the correlator is running.
 

Configuration

The configuration file connectivity.yaml defines the connectivity chain we use.  It contains a list of plug-ins used in the sample. Please note that this is a mixture of Java (JSON) and C++ samples. The source for all plug-ins is included in the samples directory as well.

The configuration file also includes an 'http' chain consisting of the following plug-ins:

  • eventMap host plug-in: This connects the chain to the correlator internals and converts the events to maps and back. The configuration also sets which channels it subscribes to.
     
  • mapperCodec: This is controlled by its configuration and renames fields or moves data from payload to metadata or vice versa. For HTTPRequest, it maps the URL from payload (from the event field) to the metadata, as the URL controls the transport rather than being the payload of the request.   
      - mapperCodec:
        apamax.weather.HTTPRequest:
          towardsTransport:
            mapFrom:
              - metadata.http.url: payload.url
              - metadata.http.id: payload.id
            defaultValue:
              - metadata.http.method: GET
 

Figure 1: mapperCodec configuration sample

  • classifierCodec: This identifies the event type being received in the HTTP response, based on the HTTP responsecode and the payload contents. 
   - classifierCodec:
        rules:
          - apamax.weather.GEOResponse:
              - metadata.http.responsecode: 200
              - payload.ip:

Figure 2: classifierCodec configuration sample

  • JSONCodec: This converts the payload into the JSON text form. This does not require any configuration; it will convert any data consisting of the map, list, string and numeric types passed through. The JSON codec leaves the metadata untouched.
     
  • httpClient transport: This sends HTTP requests and converts the HTTP response to a message. This requires no configuration; metadata controls what URL to send a request. 

931×271 20 KB

Figure 3: Example of the HTTP Chain Invoking External Services from the “Weather” Sample.
 

How it works

Using this adapter from EPL is easy; the included sample EPL application, WeatherApp.mon, sends HTTPRequest events to the "httpRequest" channel that the eventMap host plug-in is subscribed.

When EPL sends an event, it passes through the codecs to the transport:

  • eventMap: Host plug-in converts events into maps
     
  • mapperCodec: Moves the URL into the metadata "http.url" field, which the transport will later use, along with an id to correlate requests with their responses. It also sets the http.method to "GET".
     
  • classifierCodec: Just passes the message through untouched (it does not need to do anything in this direction)
     
  •  JSONCodec: Converts the now empty payload
     
  • httpClient: Transport sends the GET request to the URL specified in the metadata (from the event ‘url’ field). GET requests do not use the payload.

When the HTTP transport receives the response, it builds a message with metadata containing http.responsecode, other HTTP headers and the id that the request was sent with. The payload is a string containing the body of the HTTP response. The message is then sent towards the correlator via the plug-ins in reverse order:

  • JSONCodec: Our application sends requests to URLs that return JSON, so the JSONCodec converts the JSON response in the payload into Java objects for us.
     
  • classifierCodec: Identifies which event type the event is based on the http.responsecode and if the payload contains an "ip" entry (which it should for the GEOResponse, along with the city name, country and approximate co-ordinates). Any HTTP response code other than 200 (OK) will map to a HTTPError event.
     
  • mapperCodec: Moves the id from the metadata to the payload, and sets the correct channel.
     
  • eventMap host plugin: Converts the payload to the internal event format (internal object format, not an EPL string!). Note that the longitude, latitude and city do not need to be called out anywhere other than in the event definition in EPL; if other fields in the JSON response are desired (e.g., time_zone), simply add a field to the event definition in EPL, and it is available. (It is also possible to stash any unmapped fields into a dictionary on the event).

1101×850 46.2 KB

Figure 4: The HTTP response body and headers turn into payload and metadata and then are transformed by the codecs into the event data and event type.

Similarly, the weather is queried by sending an HTTPRequest with a URL for openweathermap.org where our location is appended to the URL in EPL. The response, which contains a "weather" field, is classified by the classifierCodec as a WeatherResponse event. This includes more complex JSON data, including an array of maps, which automatically converts to sequence<dictionary<string,string>> in EPL - again, without requiring any configuration.

This simple demonstration shows how a handful of general-purpose, re-usable plug-ins, can be used together to perform two different requests from different service providers, to build a simple application. It should be easy to use these to perform any simple HTTP GET service request, translating JSON into EPL with minimal configuration.
 

What's next for connectivity plugins?

Future releases of Apama include plans to give more power to connectivity plug-ins:

  • More powerful configuration: making it easier to split configuration across multiple files and separate simple properties to be substituted into the configuration.
     
  • Folding other existing connectivity options into the connectivity plug-ins world, such as integration with Universal Messaging as a connectivity plug-in transport, which allows use of nEventProperties or codecs such as the JSONCodec.
     
  • More re-usable Codec plug-ins that can be used in different adapter configurations such as a CSV codec and regular expression codecs. Keep an eye out on Tech Community and the Apama Community Edition site for details.
     
  • Extensions to the PySys test framework to aid testing connectivity plug-ins.
     
  • Mechanisms for dynamically creating new chain instances at runtime from EPL.
     
  • Optimizing connectivity to other Digital Business Platform products.
     
  • Integration of the connectivity plug-in feature with the existing correlator persistence mechanisms to provide the ability to do reliable messaging in a similar way to how we do for JMS.

Internally, we've found that using connectivity plug-ins is a powerful interface for rapidly developing new adapters, and include the source for several as samples in the product. We strongly encourage new and existing developers of Apama solutions to actively investigate where connectivity plug-ins could be beneficial to their projects, and to contribute to the growing Apama community with new plug-ins that are developed on GitHub or elsewhere.