Search server.log on multiple servers

,
1

Hello,
We have multiple (5) Linux clustered servers and production support is time-consuming when we have to always pull up each server and check the server.log file. We never know which server the data we’re looking for is on, so 1 by 1, we log in and search each one. Also, there are circumstances where part of what we are looking for is on one server and the other part of what we are looking for is on another server.

Does anyone out there have any tips on how to search multiple search logs at one time…say for instance, sign into just one server and type in some command that would search all 5 servers simultaneously? Or maybe, you are saving your server.log files into a database or something? Not sure, but I’m hoping you handle this differently, then having to do what we have to do.

Thanks for any ideas. webMethods version is 9.12

2

Check Command Central if you have any API’s to pull logs from each node and build a custom solution.

You can also use 3rd party log monitoring tools like Logstash.

3

You mentioned LINUX, so it should very easy to have an batch script or ALIAS that run first an SCP and pull all copies of your server.log files locally giving them meaningfull names and then run the fgrep on them.


scp   <user1@server1>:/<path to IS logs>/server.log <server1>--server.log
scp   <user2@server2>:/<path to IS logs>/server.log <server2>--server.log
scp   <user3@server3>:/<path to IS logs>/server.log <server3>--server.log
scp   <user4@server4>:/<path to IS logs>/server.log <server4>--server.log
scp   <user5@server5>:/<path to IS logs>/server.log <server5>--server.log
fgrep <what you like to see> *--server.log

It might for sure help to have SSH keys for the local user in the authorized_keys of the target system users.
I hope that helps.

Windows (or using Samba / SMB) would be similar simple with xcopy or robocopy and “grep” from an GNU package.

4

We are using Graylog for doing the same task. We have a graylog cluster and publishing the multiple IS logs into Graylog to search and alert if smthg happens.

5

Unfortunately there is no out of the box solution for this problem. I have written custom solutions in the past to achieve this using Java and creating a brand new file that has all the log file entries(from multiple servers) based on timestamp. We had about 90% success with that approach. Some times the ordering of the log entries based on timestamp doesn’t work as expected if they have same timestamp(with milliseconds). This approach also fails miserably when trace level logging is enabled.

We ran this service every 10 minutes so this information was 10 minutes late when compared with the original files. Each time it runs, it overwrites the consolidated file instead of appending to avoid writing duplicate entries.

This was a cumbersome process(too many IO read\writes) but even with its limitations client seemed to be happy with it. This process could be tweaked to do the merging async on a separate instance of IS that is dedicated to Administration work(separate from business processing).

I think a feature request can definitely be raised for this on Brainstorm.

Cheers,
Akshith