Linux kernel live debugging, how it's done and what tools are used?

Question

What are the most common and why not uncommon methods and tools used to do live debugging on the Linux kernel? I know that Linus for eg. is against this kind of debugging fo

Answer 1

KGDB + QEMU step-by-step

KGDB is a kernel subsystem that allows you to step debug the kernel itself from a host GDB.

My QEMU + Buildroot example is a good way to get a taste of it without real hardware: https://github.com/cirosantilli/linux-kernel-module-cheat/tree/1969cd6f8d30dace81d9848c6bacbb8bad9dacd8#kgdb

Pros and cons vs other methods:

• advantage vs QEMU:
  • you often don't have software emulation for your device as hardware vendors don't like to release accurate software models for their devices
  • real hardware way faster than QEMU
• advantage vs JTAG: no need for extra JTAG hardware, easier to setup
• disadvantages vs QEMU and JTAG: less visibility and more intrusive. KGDB relies on the certain parts of the kernel working to be able to communicate with the host. So e.g. it breaks down in panic, you can't view the boot sequence.

The main steps are:

1. Compile the kernel with:

   CONFIG_DEBUG_KERNEL=y
   CONFIG_DEBUG_INFO=y
   
   CONFIG_CONSOLE_POLL=y
   CONFIG_KDB_CONTINUE_CATASTROPHIC=0
   CONFIG_KDB_DEFAULT_ENABLE=0x1
   CONFIG_KDB_KEYBOARD=y
   CONFIG_KGDB=y
   CONFIG_KGDB_KDB=y
   CONFIG_KGDB_LOW_LEVEL_TRAP=y
   CONFIG_KGDB_SERIAL_CONSOLE=y
   CONFIG_KGDB_TESTS=y
   CONFIG_KGDB_TESTS_ON_BOOT=n
   CONFIG_MAGIC_SYSRQ=y
   CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1
   CONFIG_SERIAL_KGDB_NMI=n
   

   Most of those are not mandatory, but this is what I've tested.

2. Add to your QEMU command:

   -append 'kgdbwait kgdboc=ttyS0,115200' \
   -serial tcp::1234,server,nowait
   

3. Run GDB with from the root of the Linux kernel source tree with:

   gdb -ex 'file vmlinux' -ex 'target remote localhost:1234'
   

4. In GDB:

   (gdb) c
   

   and the boot should finish.

5. In QEMU:

   echo g > /proc/sysrq-trigger
   

   And GDB should break.

6. Now we are done, you can use GDB as usual:

   b sys_write
   c
   

Tested in Ubuntu 14.04.

KGDB + Raspberry Pi

The exact same setup as above almost worked on a Raspberry Pi 2, Raspbian Jessie 2016-05-27.

You just have to learn to do the QEMU steps on the Pi, which are easily Googlable:

• add the configuration options and recompile the kernel as explained at https://www.raspberrypi.org/documentation/linux/kernel/building.md There were unfortunately missing options on the default kernel build, notably no debug symbols, so the recompile is needed.

• edit cmdline.txt of the boot partition and add:

  kgdbwait kgdboc=ttyAMA0,115200
  

• connect gdb to the serial with:

  arm-linux-gnueabihf-gdb -ex 'file vmlinux' -ex 'target remote /dev/ttyUSB0'
  

  If you are not familiar with the serial, check out this: https://www.youtube.com/watch?v=da5Q7xL_OTo All you need is a cheap adapter like this one. Make sure you can get a shell through the serial to ensure that it is working before trying out KGDB.

• do:

  echo g | sudo tee /proc/sysrq-trigger
  

  from inside an SSH session, since the serial is already taken by GDB.

With this setup, I was able to put a breakpoint in sys_write, pause program execution, list source and continue.

However, sometimes when I did next in sys_write GDB just hung and printed this error message several times:

Ignoring packet error, continuing...

so I'm not sure if something is wrong with my setup, or if this is expected because of what some background process is doing in the more complex Raspbian image.

I've also been told to try and disable multiprocessing with the Linux boot options, but I haven't tried it yet.