Analyzing Linux kernel Oops or kernel panic

Classification of kernel failures

Kernel crashes can be caused by both software and hardware errors. To determine the cause, it is recommended to perform hardware tests after kernel failures (for example RAM-tests and CPU-tests)^[1].

Kernel failures are divided in:

kernel Oops
kernel Panic

Kernel Oops

A kernel Oops is an unexpected error situation in the Linux kernel. An Oops can, for example, appear due to a cross-reference of an unvalid pointer. The failure is intercepted by a so-called exception trade, the ongoing process is interrupted and an error message is written into a log file via syslog.^[2]^[3]

Kernel Panic

If an Oops appears within an interrupt trader, the kernel tries to output the Oops notification on the console. After this, the system is stopped, as there is no possibility to restore the system to a stable state. Here, the Oops can not be written into a log file via syslog anymore (as there is no disk-access possible).^[2]^[4]

Recording of a kernel Oops

In case of a Kernel Oops, the information is written into a log file via syslog - mostly to /var/log/messages- and is therefore easily accessible.

This is not possible anymore for a Kernel Panic, as hard disks or network access interrupts are necessary (that do not function anymore at this point)^[5]. In this case, there are the following opportunities^[6]:

Writing-off the notifications from the text-console: This is a bit awkward, though, it is the only possibility when the failure is unplanned. Alternatively, the notification can be recorded with a digital camera or - if available - with a screenshot of a Remote-Management-Interface (KVM-over-LAN functionality).
Additional redirecting of the console on the serial interface: all notifications are not only output on the local console, but also on a serial console. A second computer must be connected via null modem cable to the serial interface.^[7]^[8]
Using a Kdump^[9]^[10]^[11] (z.B. RHEL5) or diskdump^[12]^[13] (z.B. RHEL3, RHEL4).

Analysis of a kernel Oops

Analysis of a kernel Oops notification: detailed information on this can be found in chapter 7.2.8 Examining an Oops Report of Self-Service Linux^[14]
Analysis of a dump: a utility crash can be used for this - see also ^[15] and ^[16].

Generate kernel panic

With the help of a provoke_crash kernel module, kernel panics can be triggered manually.^[17]^[18]

References

↑ http://wiki.openvz.org/Hardware_testing
↑ ^2.0 ^2.1 http://en.opensuse.org/Bugs/Kernel#A_Classification_of_Kernel_Bugs
↑ http://en.wikipedia.org/wiki/Linux_kernel_oops
↑ http://en.wikipedia.org/wiki/Kernel_panic
↑ How do I capture an Oops? (The linux-kernel mailing list FAQ)
↑ http://www.kernel.org/doc/Documentation/oops-tracing.txt
↑ http://en.opensuse.org/openSUSE:Bugreport_kernel#Using_the_serial_console
↑ http://www.vincentverhagen.nl/2008/08/07/capturing-linux-kernel-panic-info/
↑ http://www.kernel.org/doc/Documentation/kdump/kdump.txt
↑ http://en.opensuse.org/Kdump
↑ How do I configure kexec/kdump on Red Hat Enterprise Linux 5
↑ How do I use the diskdump utility to create a vmcore file
↑ How can I save the coredump (vmcore) onto the hard disk when I get a system OOPS or a kernel panic on Red Hat Enterprise Linux 3 or 4
↑ Self-Service Linux -Mastering the Art of Problem Determination (PDF), Mark Wilding and Dan Behman, Prentice Hall International 2005, ISBN 0-13-147751-X
↑ http://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/
↑ http://en.opensuse.org/Crashdump_Debugging
↑ Crash more beautifully with Kernel-Module provoke_crash (Pro-Linux)
↑ A module for crashing the kernel (lwn.net)

More information

Author: Werner Fischer

Werner Fischer, working in the Knowledge Transfer team at Thomas-Krenn, completed his studies of Computer and Media Security at FH Hagenberg in Austria. He is a regular speaker at many conferences like LinuxTag, OSMC, OSDC, LinuxCon, and author for various IT magazines. In his spare time he enjoys playing the piano and training for a good result at the annual Linz marathon relay.

Translator: Alina Ranzinger

Alina has been working at Thomas-Krenn.AG since 2024. After her training as multilingual business assistant, she got her job as assistant of the Product Management and is responsible for the translation of texts and for the organisation of the department.

[1] ttp://wiki.openvz.org/Hardware_testing

[opensuse-classification-of-kernel-bugs-2] 2.0 ^2.1 http://en.opensuse.org/Bugs/Kernel#A_Classification_of_Kernel_Bugs

[3] ttp://en.wikipedia.org/wiki/Linux_kernel_oops

[4] ttp://en.wikipedia.org/wiki/Kernel_panic

[5] How do I capture an Oops? (The linux-kernel mailing list FAQ)

[6] ttp://www.kernel.org/doc/Documentation/oops-tracing.txt

[7] ttp://en.opensuse.org/openSUSE:Bugreport_kernel#Using_the_serial_console

[8] ttp://www.vincentverhagen.nl/2008/08/07/capturing-linux-kernel-panic-info/

[9] ttp://www.kernel.org/doc/Documentation/kdump/kdump.txt

[10] ttp://en.opensuse.org/Kdump

[11] How do I configure kexec/kdump on Red Hat Enterprise Linux 5

[12] How do I use the diskdump utility to create a vmcore file

[13] How can I save the coredump (vmcore) onto the hard disk when I get a system OOPS or a kernel panic on Red Hat Enterprise Linux 3 or 4

[14] Self-Service Linux -Mastering the Art of Problem Determination (PDF), Mark Wilding and Dan Behman, Prentice Hall International 2005, ISBN 0-13-147751-X

[15] ttp://www.redhatmagazine.com/2007/08/15/a-quick-overview-of-linux-kernel-crash-dump-analysis/

[16] ttp://en.opensuse.org/Crashdump_Debugging

[17] Crash more beautifully with Kernel-Module provoke_crash (Pro-Linux)

[18] A module for crashing the kernel (lwn.net)

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

[12]

[13]

[14]

[15]

[16]

[17]

[18]