While working on Performance issues, there is various kinds of dumps we need to get from the application or hardware environment to do the root cause of the performance issue. This paper discusses the need of the dumps and how to take these dumps on various platforms.
Note: This paper does not cover how to analyze these dumps. Java 7 and above is considered for taking dumps.
Thread Dumps:
Any Java application uses threads to process a request. Large number of concurrent users spawns more threads. When two or more threads utilize the same resources, a contention between the threads occurs. Thread dump is used to analyze thread contention issues and it provides information on the exact status of each thread and information about the call stack of each thread.
Points to remember while taking thread dumps:
- Provides information of threads at that moment of time
- One thread dump may not help
- Always take 5/10 thread dumps at an interval (say 10 sec)
- Multiple thread dumps can be analyzed to check thread status transition
- Take the thread dump when experiencing the slowness or while reproducing the issue.
1. Taking Thread Dump using jVisualVM:
If the java application is running and responsive jVisualVM can be attached to the running instance to generate the thread dump.
- jVisualVM is provided along the JDK distribution and can be found in bin directory of JDK.
- Invoke jVisualVM and connect to the java process. All the java processes in a machine are shown in Applications tab. A remote java process can also be connected to jVisualVM.
- Go to the threads tab and click on “Thread Dump” button
- The thread dump opens in a sub-tab of the application in the main window.
Note: To connect to the jVisualVM the java process (remote process) must be started with JMX port using the below command:
java -Dcom.sun.management.jmxremote.port=3333 \
-Dcom.sun.management.jmxremote.ssl=false \
-Dcom.sun.management.jmxremote.authenticate=false \
YourJavaApp
2. Taking Thread Dump using “jstack”
When an application is slow or is unresponsive, it may not be able to connect to jVisualVM. Also if automation is required to take the thread dumps, jstack is the best option.
- jstack can be found in <jre/jdk home>/bin directory. If jre/jdk is part of PATH environment variable, jstack can be invoked from command prompt.
- Find the PID of the java process.
On windows: java PID can be found in Task Manager
On Linux: use ``ps –ef | grep java``
- Run jstack on command prompt and store the output to a file:
jstack -l <PID> > ThreadDump1.txt
Note: Following shell script can be used to run and take the thread dumps automatically multiple times:
i=1
while [ $i -le 5 ]
do
echo jstack -l <PID> > ThreadDump$i
sleep 10
i=`expr $i + 1`
done
3. Taking Thread Dump using kill –SIGQUIT(3) (Only on Linux):
There can be times when application is in hang state and even jstack is now able to take the thread dumps. In that case thread dumps can be taken using kill –SIGQUIT(3).
- Find the PID of the java process.
On Linux: use “ps –ef | grep java”
- Run kill on command prompt and store the output to a file:
kill -3 <PID> > ThreadDump1.txt
4.Thread dumps can also be taken from webMethods Integration server UI
- Goto Server > Statistics > System Threads
- Click Generate JVM Thread Dump
Memory Heap Dumps:
Heap dumps contains:
- Snapshot of JVM Heap at that moment of time
- Shows live objects in heap along with references between objects
- Important in analyzing memory issues in an application
- Used to determine memory usage patterns
- Used in identifying memory leak suspects
- does not contain allocation information, i.e. what created the objects
1. Automatic generation of Heap Dumps
JVM parameter can be set while starting the Java application to take the heap dump whenever Out Of Memory (OOM) exception is encountered by application.
Set the following JVM parameter:
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dir/subdir
2.Taking Heap dump using jVisualVM
If the java application is running and responsive jVisualVM can be attached to the running instance to generate the heap dump.
- jVisualVM is provided along the JDK distribution and can be found in bin directory of JDK.
- Invoke jVisualVM and connect to the java process. All the java processes in a machine are shown in Applications tab. A remote java process can also be connected to jVisualVM.
- Go to the monitor tab and click on “Heap Dump” button
3. Taking Heap dump using jmap
When an application is slow or is unresponsive, it may not be able to connect to jVisualVM. Also if automation is required to take the heap dumps, jmap is the best option.
- jmap can be found in <jre/jdk home>/bin directory. If jre/jdk is part of PATH environment variable, jmap can be invoked from command prompt.
- Find the PID of the java process.
On windows: java PID can be found in Task Manager
On Linux: use “ps –ef | grep java”
- Run jmap on command prompt and store the output to a file:
jmap -dump:format=b,file=<heapDumpFile>.bin <PID>
4. GC logs:
Garbage collection is a complex activity done by JVM, which can affect the performance heavily. There are many GC strategies which affect the garbage collection. Its important to see if the memory allocation and JVM parameters are correctly set to size the Heap correctly in JVM. GC logs helps in collecting data for heap tuning.
Following JVM parameters should be used to enable extensive GC logging:
-Xloggc:<path_to_log_directory>/logs/gc-%t.log
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-XX:+PrintGCCause
-XX:+PrintTenuringDistribution
-XX:+UseGCLogFileRotation
-XX:NumberOfGCLogFiles=10 #Optional
-XX:GCLogFileSize=5M #Optional
5. TCP dumps:
TCP dump or network sniffer tool, saves the packets that are captured for analysis. It can be used to analyze the network performance bottlenecks and issues with virtual networks and firewalls.
Find out the interface to snoop:
tcpdump -D
Capture packets in ascii
tcpdump –A –i <interface>
Capture only number of packets
tcpdump –c 5 –i <interface>
6. CPU/Disk/Network Utilizations:
Linux:
- top # For live monitoring
- nmon # For live monitoring and logging only in HP-UX
- sar # Gives most of the Utilizations
- vmstat # vmstat <interval> for CPU stats
- netstat # netstat –a for Network stats
- iostat # iostat –c/-d For Disk I/O stats
Windows:
- Task Manager
- perfmon
Further Read: Check this article on key elements in the API response and what to do if these elements exceed or drop below a certain threshold Key elements in the API response.