The core component of Apama is known as ‘ correlator ‘ and is powered by a language called EPL (Event Processing Language) that has a syntax that would be familiar to Java programmers. The EPL is native to the core engine resembles Java-like syntax. The correlator can also be programmed by general-purpose languages like C++, Java or Python using a plug-in mechanism. These are referred to as EPL plug-ins in general, for other types of plug-ins refer to the below note.
The Apama framework also supports another feature called ‘connectivity plug-ins’. The connectivity plug-ins facilitate integrating & transforming data feed generated by external systems like MQTT, Kafka, or even plain JSON feed. The current blog post describes native API support through Java.
This blog post is an introduction to how to develop an EPL Java plug-in in the Apama environment. The post explains various development features in the Apama designer along with deployment utilities available. You can also take a look at how to write an EPL Python plug-in.
Developing an EPL plug-in using Designer
Open the ‘ Apama Designer ’ tool installed as part of Apama installation. Select new project from “ File -> New -> Apama Project “.
Name the project as needed, for e.g. “ AnEPLJavaPlugin “, and click ‘ Finish ‘.
ADDING JAVA NATURE
The Java Nature is a standard eclipse feature that allows all necessary Java JAR files accessible to the project’s java build path. The Apama Designer also comes with the “ Apama Java Nature ” feature. By adding this feature to your project you enable the use of Apama’s Java API for the Java EPL plug-in to work. To add the Apama Java Nature to an existing project select the following, in the “ Project Explorer “, right-click on the project, select “ Apama -> Add Apama Java Nature ”.
ADDING JAVA EPL PLUG-IN
The Java EPL plug-in enables writing Java code then to be called from EPL script. The Apama Designer has features to add EPL Java plug-in to an existing Apama project. This can be added in the following way, right-click on the project, select “ New -> Java EPL Plugin “.
and provide ‘ Plugin name ‘, ‘ Description ‘ and ‘ Apama Package ‘ details, click ‘ Finish ‘.
The above action opens the Java file with two annotations, @Application and @EPLPlugin . Don’t remove these, you can update details like author, version, etc…
INTEGRATING JAVA API WITH EPL MONITOR SCRIPT
Adding EPL Monitor
In order to invoke the EPI Java API, we need to have the EPL monitor script added to the project. On the Apama Designer, select the following to add a simple monitor script.
In the Apama project right click on ‘monitors’ and select ‘ New -> EPL Monitor ‘.
Provide an appropriate name to the monitor script and click ‘ Finish ‘.
Integrating the Java API with EPL script
In the Java source add a static method EPL_API() to be invoked from the EPL script, an example is shown below,
package demo;
import com.apama.epl.plugin.annotation.*;
import com.apama.jmon.annotation.*;
@Application(name = "AnEPLJavaPlugin",
author = "",
version = "1.0",
company = "",
description = "",
classpath = "")
@EPLPlugin(description = "Demo Plugin", name= "demo.DemoEPLJavaPlugin")
public class DemoEPLJavaPlugin {
public static void EPL_API() {
System.out.println("EPLPlugin message from Java");
}
}
The EPL_API() will be called from JavaIntegration.mon EPL monitor as explained below.
At the global level (outside onload or any EPL action) add the following import,
import "demo.DemoEPLJavaPlugin" as eplJavaPlugin;
The import aliases the demo Java plug-in as eplJavaPlugin to be accessible from EPL code under any action; for example, add the following code inside the onload() action,
eplJavaPlugin.EPL_API();
This is all we need to integrate the Java method to be called from an EPL script; the complete code is shown below.
monitor JavaIntegration {
import "demo.DemoEPLJavaPlugin" as eplJavaPlugin;
action onload() {
log "Loaded monitor JavaIntegration" at INFO;
eplJavaPlugin.EPL_API();
}
}
We can also add complex Java code (including third party dependencies) with Apama EPL applications in the same way.
Building the project
Inside Apama Designer we don’t need to explicitly build the project as it is automatically built, however you can run the project by selecting “ Run As -> Apama Application “. From the log messages you can see the message printed from Java method “ EPLPlugin message from Java “.
Ant build & run
The user can also use the ant build system to build the project on the command line. This can be done by exporting the ant build file from Apama Designer by right click on the Apama project, select “ export “, and search for ant , select “ Apama Ant Export ” as shown.
Click “ Next ” and provide directory path to which the build & run scripts to be written. For example, these are written to “ AnEPLJavaPluginAnt “. The generated ant script takes care of building the JAR from Java sources. We can also write the ant build file manually, however it can be error prone.
You can run ‘ ant ‘ command available in the Apama environment to build the JAR from sources following by running the binaries.
AnEPLJavaPluginAnt>ant
…
all:
BUILD SUCCESSFUL
Total time: 10 seconds
Deploying the project
Apama SDK is shipped with few utilities useful for project deployment. One such tool is engine_deploy , you can run the help to see how to use the tool. Use the following command to deploy the current project to another directory called ‘ AnEPLJavaPluginDeployed ‘.
engine_deploy -d AnEPLJavaPluginDeployed C:\SoftwareAG_10.5\Designer_Workspace\AnEPLJavaPlugin
Ensure you don’t get any error in running the above command, and a similar set of commands also works for Linux/Windows platform. The deployed project artifacts are available under ‘ AnEPLJavaPluginDeployed ‘.
RUNNING THE PROJECT ON COMMAND LINE
In order to run the EPL Java plug-in, the Apama correlator needs to know that Java code to be integrated, this can be done using -j option of the correlator. The Apama SDK comes with utilities to inject the build JAR files and monitor scripts into the running correlator. The engine_inject tool is used in this regard.
Navigate to the AnEPLJavaPluginDeployed directory, and run the following commands,
AnEPLJavaPluginDeployed>correlator -j --config config\CorrelatorConfig.yaml – This starts the correlator emitting its log to the console.
The below two commands inject the JAR file followed by the monitor script to the running correlator.
AnEPLJavaPluginDeployed>engine_inject -j -v "C:\ SoftwareAG_10.5\Designer_Workspace\AnEPLJavaPluginDeployed\AnEPLJavaPlugin java application files\AnEPLJavaPlugin.jar"
AnEPLJavaPluginDeployed>engine_inject -v "C: \SoftwareAG_10.5\Designer_Workspace\AnEPLJavaPluginDeployed\monitors\JavaIntegration.mon"
Note: It is assumed that the correlator is running on the default port, it can be overridden by using -p option including the engine_inject utility.
After inserting the EPL monitor script, the onload() action will be called as part of monitor initialization which in turn calls the EPL Java API. The correlator log can be inspected for messages emitted from Java API.
For more details on EPL Java plug-in refer to the section ‘ Writing EPL Plug-ins in Java ‘ in the Apama documentation.
A MORE COMPLEX EXAMPLE OF PARSING JSON DATA
The above example described how to call Java code from EPL without any third party dependency.
For demonstrating the next example, we will need third party Java JARs and we assume that all the required JAR files are downloaded into the project working directory. To be specific we need jackson-core-2.10.1.jar, jackson-databind-2.10.1.jar and jackson-annotations-2.10.1.jar files for this example. You can download them from Maven or other JAR repositories.
Apama EPL doesn’t have the in-built functionality of parsing JSON data in string form into an equivalent EPL dictionary data structure. In the current example, we pass JSON data in the form of a string to the Java API, and the input string originates from EPL code. The Java code further decodes the string data into a JSON structure using the Jackson parser. In the process, the JSON structure is converted into a corresponding Java map and then formatted into a string form to be interpreted by the EPL dictionary parser. Refer to the below Java code for details,
package demo;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import com.apama.epl.plugin.annotation.*;
import com.apama.event.parser.DictionaryFieldType;
import com.apama.event.parser.StringFieldType;
import com.apama.jmon.annotation.*;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.fasterxml.jackson.core.type.TypeReference;
import com.fasterxml.jackson.databind.ObjectMapper;
@Application(name = "AnEPLJavaPlugin",
author = "",
version = "1.0",
company = "",
description = "",
classpath = "")
@EPLPlugin(description = "Demo Plugin", name= "demo.DemoEPLJavaPlugin")
public class DemoEPLJavaPlugin {
public static void EPL_API() {
System.out.println("EPLPlugin message from Java");
}
private final static ObjectMapper mapper = new ObjectMapper();
private final static DictionaryFieldType<String,String> dictType = new DictionaryFieldType<>(StringFieldType.TYPE, StringFieldType.TYPE);
// Converts JSON map to a dictionary<string,string> in string form.
public static String convertJSONToMap(String input) throws JsonProcessingException, IOException {
@SuppressWarnings("unchecked")
Map<String,Object> map = (Map<String,Object>)mapper.readValue(input, new TypeReference<Object>(){});
Map<String,String> ret = new HashMap<>();
for(String k : map.keySet()) {
Object o = map.get(k);
if(o instanceof String) {
ret.put(k, (String) o);
} else {
String json = mapper.writeValueAsString(o);
ret.put(k, json);
}
}
return dictType.format(ret);
}
}
And the corresponding EPL code to pass JSON data as a string, interpret (parse) returned string as EPL dictionary is given below,
monitor JavaIntegration {
import "demo.DemoEPLJavaPlugin" as eplJavaPlugin;
action onload() {
log "Loaded monitor JavaIntegration" at INFO;
string json_data := "{\"base\":\"USD\",\"date\":\"2019-12-12\",\"time_last_updated\":1576109425,\"rates\":{\"USD\":1,\"AED\":3.672031}}";
dictionary<string,string> map := dictionary<string,string>.parse(eplJavaPlugin.convertJSONToMap(json_data));
log "Converted json " + map.toString() at INFO;
}
}
As shown in the code, we are embedding various currency exchange rates w.r.t. USD in the form of JSON data inside an EPL string (You can also consider this as external data coming via connectivity chain for processing inside correlator). The string is passed to Java API to construct a JSON object using the Jackson library. The map is converted back to a string for inter-operation between the EPL & the Java object, and the same string is parsed as a dictionary inside EPL. Without JSON parser in Java, we can’t convert the string-like JSON into EPL dictionary data structure.
CONFIGURING CLASS PATH
As we can see the parsing requires Jackson library imports. These imports can be resolved by updating build paths. The project needs to be configured to update third-party libraries like Jackson. As described earlier, download them to the local project directory, and add them to the project as explained below.
Right-click on the project, select “ Build path → Configure Build Path ”
In the next window go to the ‘ Libraries ‘ tab, and select ‘ Add External JARs ‘, navigate to the project directory (or to the directory JARs stored), and select the JAR files downloaded to the project directory.
This will resolve the import errors w.r.t. Jackson library.
These libraries can also be configured using classpath option of @Application annotation. An example,
classpath = "${env:PROJECT_HOME}/jackson-annotations-2.10.1.jar;${env:PROJECT_HOME}/jackson-core-2.10.1.jar;${env:PROJECT_HOME}/jackson-databind-2.10.1.jar"
The environment variable PROJECT_HOME should be set to the project directory or to the directory JARs stored.
Caveats
The Java code integrated with the correlator can still suffer usual concurrency-related issues. The plug-in developer needs to ensure no such bugs are introduced, if not the correlator can get into issues like deadlocks, etc…
What Next
Apama packages are available in two forms, a full installation that requires a valid license, and core installation with limited functionality. In the future, we will cover more application-specific EPL Java plug-in examples. You can find more Java plug-in examples in the samples directory apama-samples/correlator_plugin/java.