FastThread’s REST APIs are used for application monitoring, analyzing code quality during CI/CD pipeline and several other purposes. API response contains rich set of information (i.e. lot of elements).
In this article, we intend to highlight to few key elements in the API response. If values of these elements exceed or drops below certain threshold, then you might consider raising alerts/warnings or breaking the build.
1. AUTOMATIC PROBLEM DETECTION
FastThread API has intelligence to detect problems in the thread dumps automatically. Whenever FastThread detects problem, it will indicate it in the ‘problem’ element. This is THE PRIMARY ELEMENT in the response. If you encounter ‘problem’ element in the response, you are highly encouraged to generate alert/warning.
How to read ‘problem’ element?
‘problem’ element can be present in two different locations.
a. Root level
b. Every ‘threadDumpReport’ element level (Note: if you are sending 3 thread dumps in one HTTP request, then there will be 3 elements in the ‘threadDumpReport’ array.)
Here is a sample JSON response with ‘problem’ element
{
"problem": [
{
"level": "SEVERE",
"description": "9 threads are looping on...."
}
],
:
:
:
"threadDumpReport": [
{
"problem": [
{
"level": "SEVERE",
"description": "9 threads are looping on...."
}
:
:
:
},
{
"problem": [
{
"level": "SEVERE",
"description": "9 threads are looping on...."
}
:
:
:
},
}
JSON expression to get all the ‘problem’ elements from the JSON response is:
$..problem
JSON expression to get ‘problem’ element for every single thread dump is:
$.threadDumpReport[0].problem
$.threadDumpReport[1].problem
$.threadDumpReport[2].problem
Note: ‘problem’ element has a child element ‘level’. Based on the severity of the problem this element can have value: ‘SEVERE’ or ‘WARNING’. Based on this severity level you can generate appropriate alerts. i.e. if ‘level’ is ‘SEVERE’ then you may break the build. if ‘level’ is ‘WARNING’ you may just generate a warning sign in the Jenkins report.
2. VISUAL REPORT
You might want to integrate the graphical visualization of the thread dump analysis to Jenkins report or any other dashboards. API response contains ‘graphURL’ element which has a hyperlink to the visual report. This is the same visual Report that you will get when you upload the thread dumps in to the web portal. Visual report contains graphs, detailed metrics, help texts, … As old saying goes: “one picture is worth 1000 words” :-), we highly encourage you to integrate visualization of the data in your reports.
How to read ‘graphURL’ element?
‘graphURL’ element will be found in the root level of the JSON API response. Example:
{
"graphURL": "http://fastthread.io/my-thread-report.jsp?p=YXJjaGl2ZWQvMjAxOC8wNy8xMC8tLWFwaS1lMDk0YTM0ZS1jM2ViLTRjOWEtODI1NC1mMGRkMTA3MjQ1Y2MyMGI2MjA5Ny03ODg4LTQ1ZjEtYmJjMy1iNGIwZWM4YzY3OTMudHh0LS0=&channel=API"
:
:
}
JSON expression to read ‘graphURL’ element from the JSON response is:
$.graphURL
3. THREAD COUNT
Thread count can be another key metric to monitor. If thread count goes beyond a limit it can cause CPU, memory problems. Too many threads can cause ‘java.lang.OutOfMemoryError: unable to create new native thread’ in the long-running production environment.
How to read Thread Count?
For every thread dump sent in the request, there will be a corresponding ‘threadDumpReport’ array element. Every element in ‘threadDumpReport’ array will contain ‘threadCount’ element. Example:
{
:
:
"threadDumpReport": [
{
"threadCount": 100,
:
:
},
{
"threadCount": 200,
:
:
},
}
If thread count goes beyond certain threshold then also you may consider generating alerts/warning.
JSON expression to get ‘threadCount’ element for every single thread dump is:
$.threadDumpReport[0].threadCount
$.threadDumpReport[1].threadCount
$.threadDumpReport[2].threadCount
You might have a question, what should be the threshold thread count value for my application?
Take few thread dumps from your application during peak volume time. Take average of them, it should give you average thread count value. Add buffer value to it and come up with threshold value. Example: If your application’s thread count is 100 during peak volume time, you may set the thread count threshold value to be 150.
4. THREAD STATES
Application threads can in different thread states. To learn about various thread states, refer to this quick video clip. Too many RUNNABLE threads can cause CPU spike. Too many BLOCKED threads can make application unresponsive. If number of threads in a particular thread state crosses certain threshold then you may consider generating appropriate alerts/warning.
How to read Thread States?
For every thread dump sent in the request, there will be a corresponding ‘threadDumpReport’ array element. Every element in ‘threadDumpReport’ array will contain ‘threadState’ element as shown below:
{
:
:
"threadDumpReport": [
{
"threadState": [
{
"state": "WAITING",
"threadCount": 312
},
{
"state": "RUNNABLE",
"threadCount": 31
},
{
"state": "TIMED_WAITING",
"threadCount": 20
}
],
:
:
},
{
"threadState": [
{
"state": "WAITING",
"threadCount": 302
},
{
"state": "RUNNABLE",
"threadCount": 25
},
{
"state": "TIMED_WAITING",
"threadCount": 29
}
],
:
:
},
}
JSON expression to get thread count of ‘RUNNABLE’, ‘WAITING’, ‘TIMED_WAITING’, ‘BLOCKED’ thread state in first thread dump is:
$.threadDumpReport[0].threadState[?(@.state == 'RUNNABLE')].threadCount
$.threadDumpReport[0].threadState[?(@.state == 'WAITING')].threadCount
$.threadDumpReport[0].threadState[?(@.state == 'TIMED_WAITING')].threadCount
$.threadDumpReport[0].threadState[?(@.state == 'BLOCKED')].threadCount
5. THREAD GROUPS
A thread group represents a collection of threads performing similar tasks. There could be a servlet container thread group that processes all the incoming HTTP requests. There could be a JMS thread group, which handles all the JMS sending, receiving activity. There could be some sensitive other thread groups in the application as well. You might want to track those sensitive thread groups size. You don’t want their size neither to drop below a threshold nor go beyond a threshold. Less number of threads in a thread group can stall the activities. More number of threads can lead to memory, CPU problems.
How to read thread groups?
For every thread dump sent in the request, there will be a corresponding ‘threadDumpReport’ array element. Every element in ‘threadDumpReport’ array will contain ‘threadGroup’ array element as shown below:
{
:
:
"threadDumpReport": [
{
"threadGroup": [
{
"group": "ajp-bio-192.168.100.102-6093",
"threadCount": 200
},
{
"group": "DQBFacade",
"threadCount": 100
},
{
"group": "SectorwiseContractsFacade",
"threadCount": 50
},
],
:
:
},
{
"threadGroup": [
{
"group": "ajp-bio-192.168.100.102-6093",
"threadCount": 200
},
{
"group": "DQBFacade",
"threadCount": 100
},
{
"group": "SectorwiseContractsFacade",
"threadCount": 50
},
],
:
:
},
}
JSON expression to get thread count of ‘ajp-bio-192.168.100.102-6093’ thread group from the first thread dump is:
$.threadDumpReport[0].threadState[?(@.group == 'ajp-bio-192.168.100.102-6093')].threadCount