Contents
Kernel probes are a set of tools to collect Linux kernel debugging and performance information. Developers and system administrators usually use them either to debug the kernel, or to find system performance bottlenecks. The reported data can then be used to tune the system for better performance.
You can insert these probes into any kernel routine, and specify a handler to be invoked after a particular break-point is hit. The main advantage of kernel probes is that you no longer need to rebuild the kernel and reboot the system after you make changes in a probe.
To use kernel probes, you typically need to write or obtain a specific
kernel module. Such module includes both the init and
the exit function. The init function (such as
register_kprobe()) registers one or more probes,
while the exit function unregisters them. The registration function
defines where the probe will be inserted and
which handler will be called after the probe is hit.
To register or unregister a group of probes at one time, you can use
relevant
register_<
or
probe_type>probes()unregister_<
functions.
probe_type>probes()
Debugging and status messages are typically reported with the
printk kernel routine.
printk is a kernel-space equivalent of a user-space
printf routine. For more information on
printk, see Logging kernel
messages. Normally, you can view these messages by inspecting
/var/log/messages or
/var/log/syslog. Commenting, file not ready For more
information on log files, see Chapter 4, Analyzing and Managing System Log Files.
Kernel probes are fully implemented on the following architectures:
i386
x86_64 (AMD-64, EM64T)
ppc64
arm
ppc
Kernel probes are partially implemented on the following architectures:
ia64 (does not support probes on instruction
slot1)
sparc64 (return probes not yet implemented)
There are three types of kernel probes: kprobes,
jprobes, and kretprobes.
Kretprobes are sometimes referred to as return
probes. You can find vivid source code examples of all three
type of kernel probes in the
/usr/src/linux/samples/kprobes/ directory (package
kernel-source).
Kprobe can be attached to any instruction in the Linux kernel. When it is registered, it inserts a break-point at the first bytes of the probed instruction. When the processor hits this break-point, the processor registers are saved, and the processing passes to kprobes. First, a pre-handler is executed, then the probed instruction is stepped, and, finally a post-handler is executed. The control is then passed to the instruction following the probe point.
Jprobe is implemented through the kprobe mechanism. It is inserted on a
function's entry point and allows direct access to the arguments of the
function which is being probed. Its handler routine must have the same
argument list and return value as the probed function. It also has to
end by calling the jprobe_return() function.
When jprobe is hit, the processor registers are saved, and the
instruction pointer is directed to the jprobe handler routine. The
control
then passes to the handler with the same register contents as the
function being probed. Finally, the handler calls the
jprobe_return() function, and switches the
control back to the control function.
In general, you can insert multiple probes on one function. Jprobe is, however, limited to only one instance per function.
Return probes are also implemented through kprobes. When the
register_kretprobe() function is called, a
kprobe is attached to the entry of the probed function.
After hitting the probe, the Kernel probes mechanism saves the probed
function return address and calls a user-defined return handler. The
control is then passed back to the probed function.
Before you call register_kretprobe(), you need
to set a maxactive argument, which specifies
how many instances of the function can be probed at the same time. If
set too low, you will miss a certain number of probes.
Kprobe's programming interface consists of functions, which are used to register and unregister all used kernel probes, and associated probe handlers. For a more detailed description of these functions and their arguments, see the information sources in Section 6.5, “For More Information”.
register_kprobe()
Inserts a break-point on a specified address. When the break-point is
hit, the pre_handler and
post_handler are called.
register_jprobe()
Inserts a break-point in the specified address. The address has to be the address of the first instruction of the probed function. When the break-point is hit, the specified handler is run. The handler should have the same argument list and return type as the probed.
register_kretprobe()
Inserts a return probe for the specified function. When the probed function returns, a specified handler is run. This function returns 0 on success, or a negative error number on failure.
unregister_kprobe(), unregister_jprobe(), unregister_kretprobe()
Removes the specified probe. You can use it any time after the probe has been registered.
register_kprobes(), register_jprobes(), register_kretprobes()
Inserts each of the probes in the specified array.
unregister_kprobes(), unregister_jprobes(), unregister_kretprobes()
Removes each of the probes in the specified array.
disable_kprobe(), disable_jprobe(), disable_kretprobe()
Disables the specified probe temporarily.
enable_kprobe(), enable_jprobe(), enable_kretprobe()
Enables temporarily disabled probes.
With recent Linux kernels, the Kernel probes instrumentation uses the kernel debugfs interface. It helps you list all registered probes and globally switch all the probes on or off.
The list of all currently registered kprobes is in the
/sys/kernel/debug/kprobes/list file.
saturn.example.com:~ # cat /sys/kernel/debug/kprobes/list c015d71a k vfs_read+0x0 [DISABLED] c011a316 j do_fork+0x0 c03dedc5 r tcp_v4_rcv+0x0
The first column lists the address in the kernel where the probe is
inserted. The second column prints the type of the probe:
k for kprobe, j for jprobe, and
r for return probe. The third column specifies the
symbol, offset and optional module name of the probe. The following
optional columns include the status information of the probe. If the
probe is inserted on a virtual address which is not valid anymore, it is
marked with [GONE]. If the probe is temporarily
disabled, it is marked with [DISABLED].
The /sys/kernel/debug/kprobes/enabled file
represents a switch with which you can globally and forcibly turn on or
off all the registered kernel probes. To turn them off, simply enter
echo "0" > /sys/kernel/debug/kprobes/enabled
on the command line as root. To turn them on again, enter
echo "1" > /sys/kernel/debug/kprobes/enabled
Note that this way you do not change the status of the probes. If a
probe is temporarily disabled, it will not be enabled automatically but
will remain in the [DISABLED] state after entering
the latter command.
To learn more about kernel probes, look at the following sources of information:
Thorough but more technically oriented information about kernel probes
is in /usr/src/linux/Documentation/kprobes.txt
(package kenrel-source).
Examples of all three types of probes (together with related
Makefile) are in the
/usr/src/linux/samples/kprobes/ directory (package
kenrel-source).
In-depth information about Linux kernel modules and
printk kernel routine is in
The
Linux Kernel Module Programming Guide
Practical but slightly outdated information about practical use of kernel probes is in Kernel debugging with Kprobes