Hi.
The resource limit is for the entire Kubernetes pod. For Streaming Analytics there are (normally) two distinct separate processes within the pod, the JVM and the Correlator.
As mentioned in some of the previous replies and doc links, you can get the combined metrics in Prometheus format by accessing <tenantURL>/service/cep/prometheus
Examples of useful metrics:
How much memory is the pod using (in MB):
# HELP sag_apama_in_c8y_apama_ctrl_total_physical_mb Total microservice physical memory usage
# TYPE sag_apama_in_c8y_apama_ctrl_total_physical_mb gauge
sag_apama_in_c8y_apama_ctrl_total_physical_mb 662
How much memory is the correlator process using (in bytes):
# HELP sag_apama_correlator_physical_memory_bytes Physical memory usage
# TYPE sag_apama_correlator_physical_memory_bytes gauge
sag_apama_correlator_physical_memory_bytes 312905728
What are the resource limits for this whole microservice/pod, including memory limit (in MB):
# HELP sag_apama_correlator_user_streaminganalytics_microservice_metadata The user status 'streaminganalytics_microservice_metadata'
# TYPE sag_apama_correlator_user_streaminganalytics_microservice_metadata gauge
sag_apama_correlator_user_streaminganalytics_microservice_metadata{cpuCoreLimit="1.00",memoryLimitMB="4096",microserviceName="apama-ctrl-1c-4g",microserviceVersion="25.176.0"} 0
How much memory is the JVM process using (in MB):
# HELP sag_apama_in_c8y_apama_ctrl_physical_mb Java process physical memory usage
# TYPE sag_apama_in_c8y_apama_ctrl_physical_mb gauge
sag_apama_in_c8y_apama_ctrl_physical_mb 364.46875
Here my pod is using 662MB, made up of approx 364MB for the JVM and 298 for the Correlator.
As said earlier, you should ignore the virtual memory.
Other metrics from the correlator process that are potentially useful to monitor are listed in the Apama docs:
List of correlator status statistics (softwareag.com)
If you are getting alarms about hitting 90% usage (of pod limit), then there are a few ideas to check:
- That alarm described on this page: Troubleshooting and diagnostics - Cumulocity IoT documentation
- Is your output queue growing over time? (producing output faster than can be consumed)
- Is your input queue growing over time? (either the output queue is full and causing back-pressure, or for each incoming message you are trying to do too much work (or blocked on an external lookup - are you attempting to manipulate ManagedObjects at incoming Measurement rates, or similarly doing too many/inefficient remote Find requests at incoming Measurement rates?)
- Do you have a listener-leak?
Some of those can be spotted either by graphing Prometheus metrics, or by passing the correlator log file (which was in the diagnostics zip that you had) through the log-file-analyzer: Introducing the Apama Log Analyzer - Knowledge base - Cumulocity IoT - Software AG Tech Community & Forums