Analyzing diagnostics data

, ,
1

hi,

Recently we ran into memory issues on integration servers and captured diagnostics data from servers. As diagnostics data contains log files, config, services stats wondering if there is a way we can analyze this data ourselves instead of reaching out to SAG support.
Do we have any vendor tools to quickly analyze diagnostics data and pin point what is the root cause of issue?

Thanks in advance

2

Hi,

for thread dumps you can use Samurai, a free java based thread dump analyzer.

For server stats there has been a tool available in the community, but this one is no longer running due to missing runtime on recent windows versions. Additionally it will not work with memory settings above of 2GB.

Regards,
Holger

3

Hi,

Maybe it could be possible to have an excel sheet to process the stats file.

Would someone point me to the file format’s description?

Best Regards,

4

Hi Gerardo,

here are first three lines of a fresh stats.log file:


2017-04-21 16:39:23 MESZ ---( Start )---
2017-04-21 16:39:23 MESZ TotalMem         FreeMem          CurT MaxT SSNs SSNx SSNAvg   REQs REQx REQAvg   StartReq EndReq
2017-04-21 16:40:23 MESZ 000000001a2a0000 0000000010c32168 00b5 00b5 0000 0001 00000025 0000 0006 000001b3 0000005b 0000005b

All numbers are hexadecimals except for the Timestamp.
T stands for Threads and SSN stands for Sessions.

Regards,
Holger

5

hi,

For thread dumps I will download Samurai and we have some online tools to analyze but can someone advice on how we can interpret other files(like stats.log, ehache.log, service.log, session.log, terracotta-client.log) to help us identify the actual cause of high memory issues. Here are the few lines from stats.log file

2017-08-30 00:00:58 GMT 0000000179400000 00000000a442f928 0558 05a7 0003 0063 00000310 00a3 01c5 00000209 00000326 00000326

2017-08-30 00:01:58 GMT 0000000179400000 00000000982b2a38 0557 05a7 0003 0063 00000029 00a9 01c5 000003a5 000002ca 000002c4

2017-08-30 00:02:58 GMT 0000000179400000 0000000085223fb8 0558 05a7 0003 0063 000000c8 00a2 01c5 0000072f 00000986 0000098d

2017-08-30 00:03:58 GMT 0000000179400000 000000007932eaf8 0555 05a7 0003 0063 00000046 00a2 01c5 00000040 00000097 00000097
6

Hi Victor,

for the meaning of the fields in the rows of the stats.log file see my previous post.

Textual Descritption of the fields:

  • Timestamp of the entry
  • Max memory
  • Free memory
  • Current Threads
  • Max Threads
  • Sessions current
  • Sessions max
  • Session Average time
  • Requests completed current
  • Requests completed max
  • Requests Average time
  • Requests per Minute Started
  • Requests per Minute Ended

You can check for the StatsLogParser Tool in the Download section of the Community.
As mentioned before recent versions of Windows do not have the appropriate runtime installed to run this tool.
Additionally this tool can not handle log files with memory values above 2GB/2048MB (in Hexadecimal: 00000000ffffffff).

When trying to parse this file you will have to split the timestamp into 3 parts: date, time, timezone. Otherwise it is not possible to identify the fields correctly. Field separator is the space chraracter in this case.

Regards,
Holger

7

Thanks Holger! Good to know about these values. However the tool may not be right fit for my needs as we are running a 6GB memory.So am I left with only option to raise an incident with SAG now?

8

Hi Victor,

When I had to deal with such memory settings whilst the tool was working I have modified the big values to the max possible value.
In most case we were not interested in the memory consumption, but in the other values.
So this was working for me.

Unfortunately the file cannot be parsed by fixed length records as there are timezones which have different length (i.e. MET vs. MEST).

I am not sure if an incident at Software AG will help you very much here.

Lets see what Gerardo is able to achieve.

Regards,
Holger

9

Hi,

I’ve now imported the stats.log file to Excel and LibreOffice Calc (LOC is easier than Excel).

First clean the log of any extra header lines in the middle of data.

You just need to use the text import assistance specifying the space character as a separator.

You should specify:

  • date column is a date (choose appropriate format)
  • time is standard
  • all remaining columns are text

make a second worksheet with calculated values:

  • timestamp is just the sum of date and time
  • ignore the timezone
  • memory columns can be converted with HEX2DEC(RIGHT(,10)) as excel can only convert hexadecimals with up to 10 digits
  • all remaining columns can be converted with HEX2DEC

If your excel is not in english, check the correct function translation in Translator • Excel-Translator

have fun.