In the rapidly expanding ecosystems of Complex Event Processing (CEP) and Event Stream Processing (ESP, also known as Stream Processing) together known as Streaming Analytics, Apama is no longer the only choice available. The world is becoming ever more digitized, and just as the data flows grow in number and depth, so must the solutions evolve to meet these greater challenges. With both large commercial organizations and the open source community all committing their own offerings, choosing the best has never been more involved. So where does Apama Community Edition (CE) stand in the landscape of today? Let’s find out.
Although we’ve tried to make this blog as neutral and informative as possible, as engineers on the Apama R&D team there will of course be some bias in anything we write on the subject. Accordingly, rather than comparing ourselves to the various other projects out there in an invariably if subtly one sided fashion, we would rather talk about the things that excite us about the Apama platform as developers, engineers and general technology enthusiasts. As well as keeping us out of trouble, this allows you the noble reader to draw conclusions after your own analysis and testing.
Old Faithful
Apama has been in existence since the turn of the millennium, making it one of if not the first publicly available system of this nature. With much of the original team still on the project today, we’ve been thinking about Streaming Analytics for over 15 years now, entailing a wealth of experience on what works, and more importantly what doesn’t. While Apama is wildly popular in the commercial space, most SME and individual interest fall on the open-source community, most notably in several Apache Foundation projects. Apama CE offers many of the benefits of open source software, such as the free availability of use, sharing of community creations, and following from that integration with many other platforms. Additionally, Apama CE also offers the many benefits of being an actively developed commercial product, including enterprise-level tooling, constant development, and a mature, highly robust codebase. So how did Apama stay at pole position for so long?
Firstly, Apama has both incredibly low latency and very high throughput, comfortably capable of processing tens of millions of sizeable events per second through a single Correlator with latencies in the 10s of milliseconds or less. Apama started its life in Fin-Tech where latency is king, though more recently the focus has been on widening the breadth of event consumption while taking care to preserve the lightning response time; with Apama you can have your cake and eat it too. Correlators are extremely dynamic applications, it’s possible to reload events, types, and monitors and more into a hot Correlator, changing processing logic, topologies, data-sources and so on with no loss of state, service or reconfiguration. The footprint of a new Correlator is very small at only ~12MB scaling very efficiently, ideal for small devices such as in IoT applications. Not forgetting the new Connectivity API, which makes integrating with a range of technologies have both a short development time and a fast runtime; users can create and share adapters and some can even be found packaged with the CE or on this website.
EPL
A key feature in Apama’s power and success lies in our domain specific language created solely for event processing, the aptly named Event Processing Language (EPL, formerly known as MonitorScript). EPL provides a full range of processing operators, including aggregation, temporal, filtering and location operators, for example a rule such as ‘on every event C with a location field within a certain area that is preceded by 10 event Bs each with a field value above x within 10 seconds after each event A’ is easily and efficiently expressed with the language. Developing event-driven applications in traditional imperative languages can be unwieldy and verbose, however using a language designed only for this purpose is far more succinct, greatly improving code readability and development time. EPL has an intended similarity to popular languages like Java and C++, making the learning curve very slight for anyone familiar with these. If you are too attached to your existing libraries don’t despair, as you can reference them from within EPL itself, with no expensive re-implementation work. The language offers good support for both CEP and ESP paradigms, not to mention that the order of processing is absolutely guaranteed. And all of this comes in a memory-managed and multi-threaded environment, with language constructs for safely and automatically handling parallelism, streaming, and messaging across different systems within a complex deployment.
Creating complex rules and tying them to complex imperative code is quick and easy
on (all CreatePosition() as create -> PositionServiceCreated())
and not completed PositionServiceCreated()
{
processCreatePosition(create);
}
on all DeletePosition(user = user, orderKey = orderKey) as delete
and not PositionDeleted(user = user, orderKey = orderKey)
{
integer currentCount := posCounters[delete.user].decreaseCounter(delete.orderKey);
if (currentCount < 1) then
{
dvPosition.delete(delete.orderKey, delete.user);
psIface.unsubscribe(subId, cbPositionUnsubscribed);
// Route this here as can't pass the msgId - and don't want to create an interceptor
route PositionDeleted(delete.msgId, delete.user, delete.orderKey);
}
}
Under the hood of EPL is where things get really interesting. For example, a shared nothing actor model is used for the monitors; this means there is no shared state to cause contention entailing high scalability, in addition to very low garbage collection latencies as each monitor’s heap can be collected separately. The publish-subscribe approach to event routing is both fast and rich allowing loose coupling of system components, such as several Correlators talking to each other dynamically; this makes it easy to change a topology with little change to code and or config. Tight coupling between the declarative event detection and imperative action parts of the language means the application is coherent and doesn’t exist in two parts. Furthermore, not only is it easy to look for complex events but it’s also easy to create even more complex actions to go with them; you might want to fire off an action with multiple stages, further calls, and potentially listening for even more events. And last but not least, EPL also uses some quite advanced compiler technologies and optimisations, leading to performance that in some circumstances can beat even C.
The Right Tool for the Write Job
Apama has spent the majority of its existence as an enterprise application, and one thing the enterprise world demands is extensive tooling. For Apama, this comes in the form of the Software AG Designer (Designer). Designer is an Eclipse based graphical IDE, providing the usual features such as syntax colouring, code completion, and package management for both your EPL files and non-EPL plugins. Several different perspectives enable it to be used effectively by both technical users and more abstract thinkers. Designer is also a project management and launch platform, offering the ability to define, add and remove in-built and or user created EPL, Queries, community plugins, adapters, and bundles to your project from a single interface. Configuration can be performed from here too, and also ultimately launching your finished application; the Apama Runtime perspective attaches and exposes the log, terminal, and various engine management tools for your live Correlator. This perspective also houses the data-player, allowing users to measure step by step how their application behaves mid-development, or even backtest it. Finally, there’s the memory profiler for analyzing how your Correlator is using its resources. And that was just a broad overview of the Designer, please feel free to take a look at the doc for the full story.
Due to the mixed language nature of the codebase and multiple support platforms, R&D wanted a unified, generic system that could handle unit and system tests for all of the above. For this purpose, an internal Python-based testing framework was created which is still used today for Apama development. An open source framework (PySys) inspired by this internal framework is shipped with Apama and recommended for testing your own Apama applications; of course, you are free to use whatever you like for this purpose. Finally, the Correlator exposes many operations and information over a REST HTTP interface, for more information please see the doc. Please note that the REST interface, and the main correlator port in general, are not intended to be exposed on a public interface and should be exposed to internal networks only.
Final Words
The world of Streaming Analytics is currently experiencing a surge in attention, with more commercial and open source projects available than ever before. Apama is a long-standing leader in this field, offering a complete package of robust processing, very high throughput with very low latency, and broad, proven tooling. Correlators are fast, efficient, dynamic, and readily connectable with each other and external sources. The EPL language provides succinct and expressive semantics for defining both CEP and ESP rules and associated actions, with a full range of operators available. EPL is managed, multi-threaded, and guarantees ordering of processing. The tooling built around Apama is multi-faceted, including an IDE, memory profiler, data replayer, project manager, and live Correlator management tools. Additionally, a cross-platform test framework with many extensions built for Apama is freely available. So when it comes to rapidly develop a modern platform for all your Streaming Analytical needs, Apama has it all and more.
As always, please post on our forums with any questions you may have. Thanks and happy downloading!