Integrating with numerical third parties, hardware support, machine learning and more
Apama’s domain-specific programming language, EPL, has the ability to add extension points written in C++ and Java®. The 10.3 release also adds an exciting new option, writing them in Python®. This article will explore some of the new functionality that's now available.| Issue 4, 2018 |  Download PDF |
Python use cases
By providing access to Python directly from within EPL, streaming applications powered by Apama can make use of existing analytics written in Python. You can also have the sources and sinks for events use existing Python access libraries.
Hardware-access libraries
Many systems are providing libraries for accessing hardware interfaces in Python. A good example of this is the Sense HAT component for the Raspberry Pi. This has been tested with Apama and it’s possible with a very simple plug-in layer to read sensor data from Raspberry Pi devices and issue commands back to the devices connected to the Pi. Apama provides a community edition of the correlator built to run on the Raspberry Pi which can be used to test this functionality. This model can be used to provide edge analytics on small, arm-based, devices.
Numeric and scientific libraries
Python provides a rich ecosystem of libraries, such as numpi and scipi, which have extensive use in the scientific community for analyzing data. These static data analysis tools can be converted into real-time streaming applications using an Apama plug-in for Python.
Machine learning and Zementis for Nyoka
The most exciting developments recently have been in the machine learning and Artificial Intelligence (AI) space. This is another area which is heavily invested in Python integration with platforms such as Tensorflow and SKLearn. Software AG provides two stages of integration with Python ML frameworks. Another article in this edition has introduced Zementis for Nyoka provides integration between many Python machine-learning models and the standard, portable modeling language, Predictive Model Markup Language (PMML).This can then be used with
the Apama Zementis plug-in to drive analytics in a streaming analytics deployment. The Apama-Python integration lets you do rapid prototyping and development by giving you access to your models directly from EPL, including those which aren’t directly representable in PMML.
With these machine learning frameworks you can have Apama both provide data to train and update your models, but also make analytics decisions based on the results of your models. This enables predictive streaming use cases, such as predictive maintenance and advanced capital markets algorithms.
Writing EPL plug-ins in Python
The full Apama installation comes with an installation of Python
3.6.6 and Apama-specific libraries for Python for integrating with Apama.
When you write an EPL plug-in Python you need to provide a Python object which inherits from the apama.eplplugin.EPLPluginBase class and has methods decorated with the apama.eplplugin.EPLAction decorator. These methods will automatically be made available to be called from any EPL object which imports the plug-in. A simple Python plug-in might look like:
from apama.eplplugin import EPLPluginBase, EPLAction
class Plugin(EPLPluginBase):
def _init_(self, init):
super(Plugin, self)._init_(init)
@EPLAction(“action<> returns string”)
def getString(self):
return “Hello World”
This defines a plug-in with one action, which takes no arguments and returns a string.
To make it available to EPL you have to add it to your configuration file:
eplPlugins:
pluginName:
pythonFile: ${PARENT_DIR}/plugin.py
class: Plugin
Finally, you can then import and call it from EPL:
monitor Test {
import “pluginName” as plugin;
action onload() {
log plugin.getString() at INFO;
}
}
Interfacing with Python
The Python plug-in interface supports passing the full range of EPL types to and from Python, automatically converting them into appropriate Python types. Even complex nested types and event types can be passed between EPL and Python.
Arbitrary Python types, which are unknown to EPL, can be stored in the EPL "chunk" type, allowing Python objects to be created, stored with the lifetime of EPL objects and then passed back to Python at a later type to cache references to large objects which can’t be accurately represented in EPL.
Any legal EPL action signature can be declared in the decorator for a Python function. It must be a member function on the object which can accept and return compatible types and numbers to the declared arguments. A single instance of the object is created when the plug-in is first imported into the correlator and all calls to those methods from any object in EPL will be called on the same singleton instance.
Installing third-party libraries
Apama comes with a full installation of Python 3.6.6, including the pip3 package installation tool. You have two options for using third-party libraries, such as the ones described, with your plug- ins. From an Apama command prompt you can run pip3 install libraryname to install the library into your Apama installation. This will need to be done for each machine where you deploy Apama.
Alternatively you can make use of the built-in Python virtual environment system to add the libraries to your project directly. In that case you’ll need to add the path to your virtual environment to the search path for Python libraries in your configuration:
eplPlugins:
pluginName:
pythonFile: ${PARENT_DIR}/plugin.py
pythonPath:
- ${PARENT_DIR}/python-venv
class: Plugin
Summary
The Apama integration with Python, along with Zementis for Nyoka, provides tight integration with existing applications and libraries written in Python from hardware control libraries, to numeric and science libraries, to AI and machine learning. This gives exciting new options for the future of real-time analytics.
To learn more, check the Software AG Apama documentation and the Python plug-in samples available in your Apama installation.