Linux kernel live debugging, how it's done and what tools are used?

Question

What are the most common and why not uncommon methods and tools used to do live debugging on the Linux kernel? I know that Linus for eg. is against this kind of debugging fo

Answer 1

Another good tool for "live" debugging is kprobes / dynamic probes.

This lets you dynamically build little tiny modules which run when certain addresses are executed - sort of like a breakpoint.

The big advantage of them are:

1. They do not impact the system - i.e. when a location is hit - it just excecutes the code - it doesn't halt the whole kernel.
2. You don't need two different systems interconnected (target and debug) like with kgdb

It is best for doing things like hitting a breakpoint, and seeing what data values are, or checking if things have been changed/overwritten, etc. If you want to "step through code" - it doesn't do that.

Addition - 2018:

Another very powerful method is a program simply called "perf" which kind of rolls-up many tools (like Dynamic probes) and kind of replaces/depricates others (like oprofile).

In particular, the perf probe command can be used to easily create/add dynamic probes to the system, afterwhich perf record can sample the system and report info (and backtraces) when the probe is hit for reporting via perf report (or perf script). If you have good debug symbols in the kernel you can get great intel out of the system without even taking the kernel down. Do a man perf (in Google or on your system) for more info on this tool or see this great page on it:

http://www.brendangregg.com/perf.html