Linux System Failure Post-Mortem
Pages: 1, 2
When the kernel detects an unrecoverable or serious error, it prints a status report to the kernel log file. This report includes such things as the contents of the registers, the contents of the kernel stack, and a function trace of the functions being executed during the fault.
All this stuff is extremely useful -- but is in machine-readable form, and addresses vary depending on the configuration of the individual machine. So the kernel log file alone is useless when determining precisely what went wrong. This is where
ksymoops comes in.
ksymoops converts the machine-readable kernel oops report to human readable text. It relies on a correct
System.map file, which is generated as part of the kernel compilation. It also expects
klogd to handle loadable modules correctly, if appropriate.
ksymoops requires the "oops text," usually available as
Oops.file from the system logger. If this file can't be found, grab the oops text from
dmesg, or from the console -- copied by hand, if necessary.
The output of
ksymoops is a list of messages that might contain a kernel problem. Where possible,
ksymoops converts the addresses to the function name the address occurs in.
>>EIP; c0113f8c <sys_init_module+49c/4d0>
Trace; c011d3f5 <sys_mremap+295/370>
Trace; c011af5f <do_generic_file_read+5bf/5f0>
Trace; c011afe9 <file_read_actor+59/60>
Trace; c011d2bc <sys_mremap+15c/370>
Trace; c010e80f <do_sigaltstack+ff/1a0>
Trace; c0107c39 <overflow+9/c>
Trace; c0107b30 <tracesys+1c/23>
Trace; 00001000 Before first symbol
man ksymoops explains these lines in great detail, but what is important to most system administrators is the list of the function names in which problems occurred. Once you know the key function, and the functions which called it, you can make an educated guess as to the cause of the kernel error.
Be aware that the output of
ksymoops is only as good as the input -- if the
System.map file is wrong, the loadable modules don't report when they are loaded in and out, or the
object files are different from the ones present when the crash occurred,
ksymoops will produce invalid output. Run it as soon as possible after the crash, for the most accurate data -- and certainly before you change the kernel!
If you're an experienced C programmer, you might want to debug the kernel itself. Use the
ksymoops output to determine where in the kernel the problem is, then use
gdb to disassemble the offending function and debug it.
gdb> disassemble offending_function
Fixing kernel problems
You've figured out that the problem was something you can correct, perhaps a driver or a loadable module. What now?
Install any appropriate patches, check that the driver is correct -- and recompile the kernel and add it as a new lilo entry. Test the new kernel. If that doesn't correct the problem, consider reporting the problem to the linux-kernel list, or the appropriate kernel developer.
Reporting a kernel problem
If you are reporting a bug to the Linux Kernel mailing list, or to any of the linux kernel developers, post the information to email@example.com, or to the relevant developer, with the subject of "
ISSUE: one line summary from [1.]".
- One-line summary of the problem:
- Full description of the problem/report:
- Keywords (i.e., modules, networking, kernel):
- Kernel version (from
- Output of Oops.. message with symbolic information resolved using
- A small shell script or example program which triggers the problem (if possible)
- Software (use the
- Processor information (from
- Module information (from
- SCSI information (from
- Relevant sections of the system log, if any:
- Kernel configuration file and symbol map:
- Description of hardware:
- Other information that might be relevant to the problem (please look in
/procand include all information that you think to be relevant):
- Other notes, patches, fixes, workarounds:
Note that the linux-kernel FAQ states that Oops data is useless if the machine with the oops has an over-clocked CPU, or is running
vmmon from VMWare. If you have either of these, fix the problem and try to reproduce the Oops before reporting it.
Common hardware failures
If you get repeated, apparently random errors in code, your CPU fan may have died. If you're familiar enough with your equipment, you may be able to hear whether the CPU fan is running -- if not, the simplest test is to open the case and look. If the CPU fan isn't running, shut the machine down and replace the fan -- you may have saved your CPU.
If the CPU fan is running, but you're still getting random errors, suspect the RAM.
There are two common ways to test the RAM. One is to remove the suspect stick and try the machine with the other sticks of RAM, or to test the suspect stick in a known-working machine. The other is to repeatedly recompile a kernel. If you get a signal 11, the RAM is probably bad.
The final common cause of hardware failure is bad blocks on the hard drive. Use the program
badblocks to test the drive.
With a little care, and a little luck, you'll get the up-time record in your local LUG -- unless a power outage downs your machine. But I can't help you with that!
- linux-kernel FAQ
Jennifer Vesperman is the author of Essential CVS. She writes for the O'Reilly Network, the Linux Documentation Project, and occasionally Linux.Com.
Return to the Linux DevCenter.