Hi Mathias,
You should be able to hook Nagios into the webMethods IS JVM via it’s JMX support. That would be the preferred way instead of going through the IS. I’m not sure how Nagios hooks into the JVM but I’m guessing its probably like all others and you will have to either put an agent in place on the IS server and hook the JVM through that, usually it’s just a few extra settings in the server.bat/sh or it may hook in to the standard JMX interface available through Sun/Oracle JMX flags.
You can try out the JMX monitoring by hooking in JConsole(Comes with java) into the IS JVM by adding these flags to your server.bat/sh parameters:
-Dcom.sun.management.jmxremote=true -Dcom.sun.management.jmxremote.port=8585 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false
JConsole documentation heree - Using JConsole - Java SE Monitoring and Management Guide
HTH
Mark