Using Apama with Prometheus and Grafana

Apama_team · December 20, 2018, 10:00pm

In Apama 10.3 we released integration for Apama with the popular Prometheus monitoring framework. These metrics can then be easily visualized using tools such as Grafana. Apama has had the capability to be monitored via HTTP REST interfaces on Apama correlators for a number of releases now. With Apama 10.3 we added REST endpoints to the correlator that align with the specification used by Prometheus. All of the metrics exposed are of type Counter or Gauge.

In this blog post I will show you how to use Prometheus and Grafana to monitor Apama correlators running in Docker containers.

In order to demonstrate it, I have a complete working example deployed in containers which you can try out on GitHub - mjj29/apama-prometheus-grafana: A demo for Apama with Prometheus and Grafana

git clone https://github.com/mjj29/apama-prometheus-grafana.git
cd apama-prometheus-grafana.git
docker stack deploy -c docker-compose.yaml prometheus

This sample uses the Docker Store image of Apama so will require you to have logged in with an account which has access to the image. Visit softwareag/apama-correlator on Docker Hub for more details. To check that your account is enabled and logged in run:

docker pull softwareag/apama-correlator:10.3

This deployment uses a random port selected for us by Docker, so you need to interrogate it to find the port it will be available on your Docker host:

docker service inspect prometheus_grafana | grep PublishedPort

You can then access the Grafana dashboard using that port on your Docker host. Log in with the default Grafana credentials (admin / admin) and select Home → Apama Monitoring to view the sample dashboard. See below for what this should look like.

Instructions for deploying the sample in Kubernetes are available in the github README.md.

Option B – Detailed instructions to deploy with a manual setup

Alternatively, you can start with a default Prometheus and Grafana and connect them to your existing correlator instance

Connecting Prometheus to Apama

Apama has built-in Prometheus support on the standard /metrics endpoint. To configure it you need to add the following to your prometheus configuration file, substituting in the host and port that your correlator is running on:

global:
  scrape_interval: 5s
  evaluation_interval: 5s
 
scrape_configs:
  - job_name: apama
    static_configs:
      - targets: ['apamahost.mydomain.com:15903']

After starting Prometheus, you should be able to access its web UI and see the correlator connected under Status→Targets in the menu.

Connecting Grafana to Prometheus

Log into your Grafana server’s UI. If you’ve just set it up, the default username and password are both ‘admin’. Select Configuration→Data Sources and Add data source. It should be a Prometheus data source and you want to add it with the URL to your Prometheus server, which will be something like http://prometheusserver.mydomain.com:9090 . It will be the URL you used to check the connection to Apama in the previous step. The remaining defaults should be correct, but you particularly need to ensure that Access it set to Server when running inside containers.

Select ‘Save and Test’ and Grafana should tell you that the Data source is working. You can now add a dashboard.

Creating a dashboard in Grafana

From the side bar in Grafana select the four squares icon, and go to ‘Manage’. Here you can select the ‘+ Dashboard’ icon to create a new dashboard. This gives you the pane to start creating widgets which are fed from Apama. We’re going to demonstrate creating a graph, so select ‘Graph’ from the palette:

apama_promotheus_grafana_2

This will give you a default graph with a random walk. Click on the Panel Title and select Edit. There are several tabs to configure, we’re interested in General and Metrics. On General you can set the title for the graph. On Metrics we’ll add the data in several steps:

Change Data Source from ‘default’ to ‘Prometheus’
In the query by A, type sag_apama_correlator_physical_memory_bytes . This should cause a line to appear on the graph. You can refresh the display with the refresh button in the top right
In the Legend format field type Physical Memory to set the legend for the graph.
click Add Query by B and type sag_apama_correlator_virtual_memory_bytes . This will cause a second line to appear on the graph.
Put Virtual Memory in the legend. Where the legend is below the graph, click on the line next to virtual memory and move it to the right Y-axis to have the lines display better on the graph

This should now look like:

When you’re typing in the query boxes you can see the functions and available metrics for Apama. One useful metric which is used in the sample is the rate of messages received, which you can write as:

irate(sag_apama_correlator_input_total[10m])

Full sample dashboard

The sample dashboard which is provided in the github project looks like this:

Adding custom application metrics

As well as the standard metrics which Apama itself exposes you can use the EPL user-defined status mechanism to expose further metrics to Prometheus. Any user-defined status can be exposed over Prometheus, provided the status name is acceptable and the value is lexically equivalent to a number. All user-defined status names first go through a simple escaping scheme where each comma (,), period (.) and hyphen (-) character is replaced by an underscore (_), and then checked against the Prometheus metric name regex. If this passes, the name is considered acceptable. When choosing names for your custom metrics, try to follow the Prometheus Best Practices, especially choosing suffixes for base units.

Custom application metrics will be visible when creating queries in Grafana with the prefix sag_apama_correlator_user_ .