Javacores? Whats the big deal. Well, javacores is a great diagnostic feature that allows you to see exactly what is happening in a JVM and Application at any given point in time. This can be extremely useful when troubleshooting issue such as performance, Out of Memory, Crash and Hangs. Javacore is a text file created by the JVM that contains detailed information on the state of the JVM, specifically the thread status and what code they are executing. The purpose of this blog post is to give a brief introduction to javacores and what you can find in them. In future blog post, I will expand upon the way we can use javacores to troubleshoot multiple types of issues.
By default, javacores are created in the WebSphere profile directory and will log an entry to the native_stderr.log indicated that it was taken and the location it was saved. You can create javacores manually using wsadmin utility or issuing a kill -3 on the process id. I would encourage you generate a javacore now so you can follow along and understand the most important parts of a javacore.
Javacores can be generated manually or automatically.
Automatically by the JVM under certain circumstances ( Crash, OutOfMemory)
Manually, when requested
• On Unix and Linux systems the most common way is using ‘kill -3 <pid>
I wanted to reference the most used MustGather documents used when troubleshooting different types of issues. These will contain the information to validate your environment is configured to automatically collect javacores and or provide scripts that will taken them for you. If you have ever worked with IBM Support on one of the issues I mentioned above, you were probably asked to get multiple javacores 30 seconds apart. This is a critical step when troubleshooting as a single javacore does not capture historical trends, therefore you need multiple javacores at different times, to establish whether the observed results are typical or not.
By default, the script will take a javacore 2 minutes apar for 6 minutes. To get javacores generated 30 seconds apart, you will need to edit the perf script download from MustGathers. So for example, edit the linperf.sh and look for the following entry:
SCRIPT_SPAN=240 # How long the whole script should take.
JAVACORE_INTERVAL=120 # How often javacores should be taken.
Change the JAVACORE_INTERVAL to be 30 so it will produce a javacore every 30 seconds until he script span is hit.
Javacore Breakdown (the most commonly used areas when troubleshooting) :
The most important areas within a javacore that can be helpful when analyzing javacores are:
•Signal Information: You can find the Signal Information at the beginning of a javacore. This is useful to understand the reason a javacore was generated. If you search for SIGNINFO, it will show you if the javacore was generated manually by a user or if there was a specific condition that caused the javacore to be created. If the javacore was created manually, the signal information will be “user”. Here are some examples of common signal:
SIGQUIT = quit, generated from terminal special char. This signal usually does not terminate the running process.
SIGSEGV = segmentation violation
SIGKILL = kill (cannot be caught or ignored).
SIGTERM = software termination signal
• Monitors: The monitor information gives information on what synchronization locks are held by which thread. It can be found by searching for “LOCKS subcomponent”. The monitor section has two types: internal and application. For each monitor, this section will show the name of the monitor, the thread which owns the monitor(if any), and a list of threads waiting to enter or be notified. If it shows unowned, this may mean an internal lock or waiting on a notify. When it is owned, it will display the thread information that owns the lock. You can use this information to find out what the thread is doing and why it is blocking. So let's take for example if we see the following entries in the monitoring section of a javacore:
3LKMONOBJECT com/ibm/ws/classloader/CompoundClassLoader@0x0000000042677F08/0x0000000042677F14: owner "WebContainer : 4" (0x00000000330AF700), entry count
3LKWAITERQ Waiting to enter:
3LKWAITER "WebContainer : 1" (0x0000000032F6D700)
3LKWAITER "WebContainer : 2" (0x0000000033022C00)
3LKWAITER "WebContainer : 8" (0x000000003364C300)
3LKWAITER "WebContainer : 9" (0x000000003371B600)
3LKWAITER "WebContainer : 10" (0x0000000033779500)
3LKWAITER "WebContainer : 41" (0x0000000033D93F00)
3LKWAITER "WebContainer : 43" (0x0000000033684000)
3LKWAITER "WebContainer : 44" (0x0000000033499F00)
3LKWAITER "WebContainer : 46" (0x0000000033E13E00)
We can see that WebContainer 4 is blocker and the other threads are waiting for it to complete. Next, you would want to know exactly what WebContainer 4 is doing, so you can scroll down to the Thread section and review the stack information to see what code is being executed.
• Threads: After looking at the monitor information, you can look at the thread dump section. You can search for “THREADS”. This section will contain the call stack for each thread in the JVM. Usually the most important threads to focus on are the WebContainer threads. When looking at the threads, it will show you the state. This is important to understand what the thread was doing at the time the javacore was taken.
The thread stack shows the java calls that have been made to get the thread to its current state. The first line in the java stack is the last method call that was made before the javacore is created. So if we look at an thread stack that I pulled from a random javacore. We can break down exactly how read the stack and the flow. See following image that shows the flow in the stack, it reads top down, so each method was executed by the call directly underneath. In this example, we can see that the bottleneck is at the database layer as the thread is waiting for response from query being executed.
Click image below for larger view:
Available Tools to review javacore:
Though you can read the javacore with a text editor, there are tools available to help with review and analysis of javacores. I find the following tool is useful when reviewing javacores especially when you are trying to compare threads in multiple javacores. Thread and Monitor Dump Analyzer - is a GUI interface that allows identification of hangs, deadlocks, resource contention, and bottlenecks in javacores. It can be obtained from following site: