Setting logging destination to database in Kubernetes

*MSR version/fix10.15: *

Hello experts, We are running MSR in a Kubernetes cluster and would like to point our logging destinations to database instead of file. we have created a custom image and are able to inject files into the final docker image. Can these settings be done from application.properties? If yes what would the key look like or else is there a separate file that we should inject to the image?

Be very cautious with what you’re doing here:
If your queue provider (in your logger conf) is set to internal, then you will loose log content if Kubernetes needs to recreate the pod (for instance because it’s unhealthy.)
If your queue provider is UM and you’ve correctly implemented durability at UM level, then you’re on the safe side.

The Kubernetes way of dealing with these logs is to keep them in files which are persisted at worker node level (they survive to pod deletion), and then you configure a log collector (such as FluentD) that sends the content of these files to a central database like Elastic Search or Splunk.

All this having been written, the file where these settings are stored is: /opt/softwareag/IntegrationServer/config/auditing/AuditConfig.xml

It does not look like these settings are managed in the MSR properties.
However you should be able to inject this file into your containers with ease (for instance by placing its content in a config map.)

One thing you can configure in the MSR properties is the JDBC pools.
Here’s an example for Postgres:

tjdbc.wmdb.dbURL=jdbc:wm:postgresql://<server-name>:5432;databaseName=wm
jdbc.wmdb.userid=postgres
jdbc.wmdb.password=$secret{WM_DB_PASSWORD}
jdbc.wmdb.maxConns=50
jdbc.wmdb.driverAlias=DataDirect Connect JDBC PostgreSQL Driver
jdbcfunc.ISCoreAudit.connPoolAlias=wmdb
jdbcfunc.ISInternal.connPoolAlias=wmdb
jdbcfunc.ProcessAudit.connPoolAlias=wmdb

The first part (jdbc.wmdb) defines a pool alias (the database connection password is injected using a K8S secret here.)
The second part defines the functional roles associated to this alias.

What is the reason/use-case for this?

Hi Stephane, Thanks for the suggestion. Yes, I have tried to inject this file already, but the file has a tag in the beginning called server id like this softwareag-msr-67768f6f97-v6htf:5555. I can’t set this tag before injecting because I wouldn’t know the server id in advance as it’s a stateless implementation and setting this to empty causes the startup to fail it gives an error saying the file format wasn’t recognized.
About the setting up of jdbc pool alias we have that configured and works well. Our JDBC pools are pointing to the pool alias correctly.

Hi Christoph, we want to enable service and document resubmission in MSR without using MWS and setting logging destinations to database is a prerequisite.

Alright, so you don’t need to change the logging configuration for this.

What you need:

  1. a MSR with WmMonitor installed
  2. a wM DB that is initialized with the appropriate schemas
  3. a JDBC pool alias to connect to this wM DB
  4. functional roles associated to this JDBC pool alias
  5. service auditing activated using properties

Regarding #1, this isn’t possible yet with the image provided @ containers.softwareag.com, you need to build a custom image following this: Reverb

2 shouldn’t be a problem.
#3 has already been done by you.
Regarding #4, you can use these 3 properties (change wmdb with the name of your JDBC pool alias):

jdbcfunc.ISCoreAudit.connPoolAlias=wmdb
jdbcfunc.ISInternal.connPoolAlias=wmdb
jdbcfunc.ProcessAudit.connPoolAlias=wmdb

Note: I’m not sure ProcessAudit is still needed here.

Regarding #5, you could use the following properties to activate auditing on all top level services, except those provided by wM:

settings.watt.server.audit.service.include=$toplevel
settings.watt.server.audit.service.exclude=$wmservices

You may have issues connecting to the MSR admin UI, in which case check this thread: https://tech.forums.softwareag.com/t/running-msr-with-multiple-replicas-in-kubernetes/294385

Please have a look at XY problem - Wikipedia

Hi Stephane,
Thanks for the answer.

  1. a MSR with WmMonitor installed - yes, we have that created using a custom image like suggested.

  2. a wM DB that is initialized with the appropriate schemas - yes, we used db configurator to create schemas

  3. a JDBC pool alias to connect to this wM DB - yes, we have this as well-done using application.properties.

  4. functional roles associated to this JDBC pool alias - also done using application.properties

  5. service auditing activated using properties - instead of properties our services have auditing setting enabled from service properties
    image

Still, we can only see the monitoring screen in the MSR GUI where the services with auditing enabled are executed but no instances are visible. As per this [auditing guide]( Requirements for Service Monitoring (softwareag.com)) setting service log to an external database is a must. Hence the question for changing logging configuration and point these loggers to the database instead of file.