This article will explain how to configure Azure monitoring of webMethods JVMs using CollectD and the JMX plugin.
Supported Version: 9.12 onwards
Summary
The Azure platform provides its own centralized logging and monitoring facility, known as Azure Monitor, for both events and run-time metrics. One feature of Azure Monitor is Application Insights, which provides rich dashboarding, along with a configurable alerting mechanism. By default, statistics are logged at the tenant and VM level. Furthermore, a number of APIs are available for applications, including Node.js and Java.
While standard VM-level measurements are available by default, it is often useful to collect more detailed statistics directly from the guest OS and/or individual JVMs, especially in cases where multiple webMethods products are co-hosted on the same VM. Since Windows has a native extension, this document only covers Linux.
To begin collecting metrics from Linux and Linux-hosted JVMs, you will need to install CollectD and its Application Insights plug-in. CollectD is an open-source data collector with facilities for many common system and network statistics. There is also a JMX plugin for collecting detailed JVM-level data. This article covers the basic steps for installing and configuring CollectD and the plugins for Application Insights, Java and JMX.
This article presumes the reader is comfortable with Linux and webMethods product administration, as well as Java and JMX. Note that the package and jar file version numbers may differ in your installation.
Reference: https://docs.microsoft.com/en-us/azure/azure-monitor/overview
Setup for Application Insights monitoring
Prerequisites
- Microsoft Azure account and Application Insights instrumentation key.
- For Java applications, make sure Microsoft CA certificates are in the trust store. Otherwise, service fails with SSL errors only visible in wrapper.log as events are buffered and sent asynchronously.
- Linux sudo access is required to complete the required setup tasks.
- Reference: https://docs.microsoft.com/en-us/azure/azure-monitor/app/java-collectd
Installation and Configuration
1. If desired, add new JMX ports for Azure Monitor. This example use ports 32768 and 32767 for external and RMI connections respectively. For simplicity, neither SSL nor authentication is used, although both are supported. Add the following lines to the custom_wrapper.conf of desired runtime and restart to enable JMX:
wrapper.java.additional.800=-Dcom.sun.management.jmxremote.port=32768
wrapper.java.additional.801=-Dcom.sun.management.jmxremote.authenticate=false
wrapper.java.additional.802=-Dcom.sun.management.jmxremote.ssl=false
wrapper.java.additional.803=-Dcom.sun.management.jmxremote.rmi.port=32767
2. Download and install the Extra Packages for Enterprise Linux (EPEL) repository:
$cd /tmp
$sudo wget https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
$sudo yum install epel-release-latest-7.noarch.rpm
3. Install collectd:
$sudo yum install collectd
4. Install Java and JMX plugins:
$sudo yum install collectd-java
$sudo yum install collectd-generic-jmx
5. Verify packages installed correctly:
$sudo yum list installed collectd*
Loaded plugins: langpacks, product-id, search-disabled-repos
Installed Packages
collectd.x86_64 5.8.1-1.el7 @epel
collectd-generic-jmx.x86_64 5.8.1-1.el7 @epel
collectd-java.x86_64 5.8.1-1.el7 @epel
6. Download Application Insights plugin :
https://github.com/Microsoft/ApplicationInsights-Java/releases/tag/2.3.1
7. Download the JAXB libraries:
https://mvnrepository.com/artifact/javax.xml.bind/jaxb-api/2.4.0-b180830.0359
8. Copy the JARs to /usr/share/collectd/java.
9. Assign ownership of the JARs to root:
$chown root:root applicationinsights-collectd-2.3.1.jar jaxb-api-2.4.0-b180830.0359.jar
10. The CollectD JMX adapter uses a custom data type, jmx_memory. The types.db.custom file has a single entry for this:
jmx_memory value:GAUGE:0:U
Upload types.db.custom to /usr/share/collectd/ and change ownership to root:
chown root:root types.db.custom
11. Replace /etc/collectd.conf with customized collectd.conf and change ownership to root:
$chown root:root collectd.conf
12. Edit /etc/collectd.conf to customize for local environment:
Edit the Host parameter in the provided collectd.conf file to the logical name of the instance, and update the port number to the eternal JMX port if necessary. Where multiple runtimes are running on the same VM, add a Connection block for each with a unique Host parameter:
The JMX plugin requires two ports, one for external connections and one for the RMI registry. This example assumes a logical server name of DEV-IS, with uses 32768 (external) and 32767 (RMI).
$sudo vi /etc/collectd.conf
…
<Plugin "GenericJMX">
...
<Connection>
Host "DEV_IS"
ServiceURL "service:jmx:rmi:///jndi/rmi://localhost:32768/jmxrmi"
Collect "garbage_collector"
Collect "memory-heap"
Collect "memory-nonheap"
# Collect "memory_pool"
</Connection>
</Plugin>
…
Edit Application Insights instrumentation key and replace with your key:
LoadPlugin "com.microsoft.applicationinsights.collectd.ApplicationInsightsWriter"
<Plugin ApplicationInsightsWriter>
InstrumentationKey "x99xx99x-9x9x-9999-9x99-99x9x99xx9"
</Plugin>
13. Fix Java symbolic link to point to your JVM libraries. Depending on the environment, the Java location may vary. Example:
$sudo ln -sf /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre/lib/amd64/server/libjvm.so /usr/lib64/libjvm.so
14. Once any errors are resolved, open JMX ports in firewall:
$sudo firewall-cmd --zone=public --add-port=32768/tcp
$sudo firewall-cmd --zone=public --add-port=32767/tcp
15. Test collectd starts and that your desired plugins load:
$sudo collectd
plugin_load: plugin "cpu" successfully loaded.
plugin_load: plugin "df" successfully loaded.
plugin_load: plugin "java" successfully loaded.
plugin_load: plugin "memory" successfully loaded.
If any errors are encountered starting collectd, enable logging in /etc/collectd.conf:
#Uncomment below to enable debug logging of collectd
#LoadPlugin logfile
#<Plugin logfile>
# LogLevel info
# File "/tmp/collectd.log"
# Timestamp true
# PrintSeverity false
#</Plugin>
For debugging, you can start collectd using the “-f” flag to prevent forking to a background process:
$collectd –f
You can also use the “-t” flag to test your configuration file parses properly:
$collectd –t
16. Once collectd starts properly, expect your custom metrics to appear in Azure Log Analytics after 5-10 minutes.
You also add the metrics to custom dashboards and/or configure alerts.
References:
https://collectd.org/wiki/index.php/Main_Page
https://collectd.org/wiki/index.php/Plugin:GenericJMX