The latest version of Apama Community Edition has arrived bringing Python script execution, Prometheus monitoring and performance improvements.
Python script execution
Apama developers can now execute Python scripts within the correlator and take advantage of the many third-party Python libraries available.
Similar to how you can create EPL plug-ins for Java, you can now create EPL plug-ins for Python. This makes it as easy as follows to call Python methods from within your Apama EPL application:
monitor m {
import "pluginname" as myplugin;
action onload() {
boolean success := myplugin.eplaction1(42, "Hello World", 3.14);
}
}
}
Apama customers often have Python scripts to pre-process their data prior to analyzing it with Apama. In addition, customers also use popular Python analytical libraries such as scikit-learn, numpy, seaborn, pandas and keras for machine learning and scientific computing. You can now tightly integrate your Python scripts into the Apama correlator to work as part of your streaming analytics workflow.
Please also refer to Zementis Nyoka, which provides an open source method for creating standards-based PMML from Python existing models – you can find Nyoka on GitHub.
Stay tuned over the coming weeks for a series of developer blogs exploring this new functionality in detail.
PySys testing framework upgrade
The PySys testing framework has been updated and includes a set of new capabilities:
New features in 1.3 include:
- Tests can now be written using Python 3
- Easier to run tests in parallel and reproduce race conditions
- Coloring of console output indicating passes and fails etc.
- New framework to help record performance (for example, throughput and latency)
- Logging of progress giving you feedback during long test runs
Note: This version of Apama provides Python 3.6.6 instead of 2.7 that was provided with previous releases. You may wish to upgrade your existing PySys Python 2 test scripts.
Monitoring using Prometheus
You can now use Prometheus, the prevalent open source monitoring and alerting toolkit, to monitor your Apama correlators.
Apama exposes correlator statistics to Prometheus over HTTP using the standard /metrics endpoint. The statistics produced include user-defined statuses from EPL and connectivity plug-ins which are automatically exposed over the same endpoint.
A common use of Prometheus is to integrate with visualization tools like Grafana to give you quick and easy ways to visualize the status of your deployments.
Compiled runtime is even faster
The native EPL compiler has been updated, giving your Apama EPL a signficant speed increase. Apama provides the native EPL compiler to compile your EPL application into architecture-specific machine code. For many tasks this is faster than Java and for processing-intensive tasks this can even be faster than C.
In the Black Scholes trading algorithm, using the new native EPL compilation option gives performance measurements that are 37% faster than using the option in the previous version of Apama.
Docker enhancements
Building Docker images for your Software AG Designer projects has been made more straightforward with the inclusion of a menu command (Apama > Add Docker Support) to create a suitable Dockerfile in your project folder.
The Apama correlator image on Docker Store includes the Cumulocity IoT transport, allowing the image to be directly used by applications connecting with Cumulocity IoT.
A new Apama builder image that helps you build your own application images using the Apama correlator base image is now available on Docker Store. This builder image includes deployment-related tooling such as engine_deploy and PySys, and is intended to be used in a Docker multi-stage build.
Docker and Kubernetes can now use secrets to securely pass credentials to the correlator.
Cumulocity IoT
Apama is now integrated into Cumulocity IoT and Cumulocity IoT Edge to allow you to use Apama’s streaming analytics within Software AG’s IoT platform.
If your application is reasonably straightforward and contained within a single EPL file, then you can upload your Apama applications into Cumulocity IoT using the Administration “Own Applications” page and selecting “Upload custom Apama rule”.
Alternatively, if your Apama application is more complex (for example, if it contains plug-ins, or has logic spread over multiple EPL files), then you can create your own custom microservice containing your Apama project. You can use the Apama tooling and documentation to help you create a Docker image for your application and then use the Cumulocity IoT guide “Microservice package and deploy” to help you pack and deploy your Docker image as a Cumulocity IoT microservice.
Integration with Cumulocity IoT is through the inclusion of the Cumulocity Client connectivity bundle which includes a connectivity plug-in allowing you to send and receive measurements, events, operations and alarms to and from Cumulocity IoT.
EPL enhancements
The Apama programming language, EPL, has a few improvements:
- New
empty()action on the any type. Returns true if the any variable does not contain a value or object. - New
hash()action on all built-in types. Returns an integer hash value which could be used with%(integral remainder) to partition incoming data. - Dictionary sizes are now 64-bit, that is, they can be larger than the previous 32-bit size.
Miscellaneous
There are many other notable improvements to Apama:
- The MQTT transport has improved security, allowing you to use client-side TLS.
- The Terracotta TCStore driver has improved security by supporting TLS for secure connections to Terracotta.
- The Classifier codec, which lets you assign incoming messages to Apama event types, now supports regular expressions. This makes it easier to identify which event types to use from the message payloads.
- Working with JSON is easier. You can use the new JSON EPL plug-in for doing adhoc conversion between EPL objects and JSON strings. There is already an existing JSON codec connectivity plug-in to help you convert from JSON in your message to Apama event fields.
- A new Automatic onApplicationInitialized bundle automatically indicates your application is ready to receive messages by calling
onApplicationInitializedafter injection has been completed. If you have a more advanced application, when you are waiting for a memory store or database to be prepared, then you should callonApplicationInitializedexplicity. - Apama dashboards now include an HTML implementation of the pie graph to help improve performance by moving rendering from the server to the client.
- jemalloc is provided as an alternative memory allocator on Linux. This frees memory more quickly, but with the cost of reduced performance.
- The correlator REST API helps you identify how memory is being used by providing additional statistics about memory usage, for example, physicalMemoryMB, swapPagesRead, jvmMemoryAllUsedMB.
- Guidelines have been provided to help identify where personal data may be used by Apama applications. If you are affected by privacy regulation such as the European GDPR, then these guidelines will help you identify where personal data may be stored.
10.3 full commercial edition
In addition to the above new features in Apama Community Edition, the full commercial edition of Apama 10.3 includes improved integration with other Software AG products, enhanced web-based management and monitoring capabilities from Command Central, and additional support for Capital Markets. Please refer to the release notes for the complete set of new features in Apama 10.3.
Find out more
Apama 10.3 Community Edition is available now for you to download. More information about the new capabilities mentioned here and numerous other smaller improvements are described in the release notes and product documentation available with the downloads.
The Software AG Apama team is extremely proud of this new version. Please give Apama 10.3 a try and let us know how you get on!