Introduction to Python Plug-ins

Apama_team · November 11, 2018, 10:00pm

In Apama 10.3 we introduced a new way to create extensions to EPL (the Apama language). In addition to writing extension points (plug-ins) to EPL in C++ and Java, it is now possible to also create them in Python.

The new interface to Python is a fully-fledged mapping between EPL and Python types allowing you to expose Python methods to be called directly from EPL with complex type arguments returning values and throwing exceptions directly back to EPL. This can be used to provide access to existing code bases and libraries or implement complex logic outside of EPL. You can integrate with third-party libraries such as machine-learning frameworks like Tensorflow or hardware interface libraries like RasPI Sense Hat.

Prerequisites

Apama 10.3 ships with a copy of Python 3.6.6. This is available in the full community installers, the Docker images published via Docker Hub and the full commercial installers. If you install with the community core installation we do not include a copy of Python. You will need to provide a separate installation of Python 3.6. Configuration of Apama to use an external installation of Python is given in the README.txt in the core installation and will be covered in a future blog post.

Creating a Python plug-in

To create a Python plug-in you must create a Python class whose methods will be made available to EPL. It must inherit from the apama.eplplugin.EPLPluginBase class and will use the apama.eplplugin.EPLAction decorator to declare which methods will be made available to EPL and in what manner.

from apama.eplplugin import EPLPluginBase, EPLAction
 
class EPLPlugin(EPLPluginBase):
   def __init__(self, init):
      super(EPLPlugin, self).__init__(init)
   @EPLAction("action<> returns string")
   def helloWorld(self):
      return "Hello World"

The argument to the EPLAction decorator is the type that the action will be exported with in EPL. The format of the argument is the same as the syntax for declaring action variables in EPL.

Using a Python plug-in

Using the plug-in from within EPL requires two steps. First you must configure the plug-in in the YAML configuration file, giving the path to the plug-in source file (use ${PARENT_DIR} for a relative path compared to the configuration file) and the name of the class within that file:

eplPlugins:
   helloWorldPlugin:
      pythonFile: ${PARENT_DIR}/plugin.py
      class: EPLPlugin

You can then start the correlator using that configuration file:

correlator --config pluginConfig.yaml

Once you have exposed the plug-in in the configuration the second step is to import it into some EPL:

monitor CallPlugin
{
   import "helloWorldPlugin" as pythonPlugin;
   action onload()
   {
      string rv := pythonPlugin.helloWorld();
      log rv at INFO;
   }
}

The name you give to import should match the one in the configuration file.

Finally, inject the monitor file:

engine_inject CallPlugin.mon

2018` `-` `08` `-` `31` `13` `:` `32` `:` `28.064` `INFO  [` `139859535492864` `] - CallPlugin [` `1` `] Hello World

Using third-party libraries with Python plug-ins

The Python installation within an Apama installation comes with all the standard Python core libraries. However, you may wish to use additional libraries. There are two main ways to do this. The simplest is to add those libraries to your Python installation. You can do this with the pip3 program which is provided, either by installing from a wheel file, or directly from the internet. The other method is to add those libraries to an external location, possibly within your project. We recommend using the Python virtual environment support, which lets you again use tools like pip to install into that environment. If you have any external libraries then you can provide those when you load your plug-in by providing additional elements to your Python path:

eplPlugins:
   machineLearningPlugin:
      pythonFile: ${PARENT_DIR}/aiplugin.py
      class: AIPlugin
      pythonPath:
         - ${PARENT_DIR}/../tensorflow-venv/Lib/site-packages

Caveats

It’s worth noting that although we’re embedding Python into the correlator which can execute on multiple contexts simultaneously and call into either the same or multiple Python plug-ins from mulitple contexts, the Python interpreter is still limited by the Global Interpreter Lock (GIL). This means that you should be careful how you use the plug-in interface as it can limit your parallelism.

Python plug-ins must be written in Python 3 with Python-3-compatible libraries. We do not support Python 2.