How to use kgdb

From RidgeRun Developer Connection
Jump to: navigation, search

I have found kgdb a great way to learn about various parts of the kernel, such as how driver probing works. That said, others have strong opinions:

Do _not_ use kgdb - which is the modus operandi of every sane kernel developer on the planet.

quote from Thomas Gleixner (kernel contributions include High Resolution Timers, Realtime-Linux with Ingo Molnar, and NAND flash driver), who clearly thinks kgdb isn't needed.

Overview of key steps

Commands to be run on the Ubuntu host have a yellow background. Commands to be run on the ARM target have an aqua background.

Configure kernel for kgdb support

Add to kernel command line in order to stop kernel boot and attach kgdb

kgdboc=ttyS0,115200 kgdbwait

 If you say Y here the resulting kernel image will include                                                                       
 debugging info resulting in a larger kernel image.                                                                               
 This adds debug symbols to the kernel and modules (gcc -g), and                                                                  
 is needed if you intend to use kernel crashdump or binary object                                                              
 tools like crash, kgdb, LKCD, gdb, etc on the kernel.                                                                       
 Say Y here only if you plan to debug the kernel.                                                                          
 If unsure, say N.                                                                                                        
 Symbol: DEBUG_INFO [=y]                                                                                                            
 Prompt: Compile the kernel with debug info                                                                                           
   Defined at lib/Kconfig.debug:575                                                                                                       
   Depends on: DEBUG_KERNEL                                                                                                               
     -> Kernel configuration                                                                                                                
       -> Kernel hacking               
   menuconfig KGDB
          bool "KGDB: kernel debugging with remote gdb"
          depends on HAVE_ARCH_KGDB
          depends on DEBUG_KERNEL && EXPERIMENTAL
            If you say Y here, it will be possible to remotely debug the
            kernel using gdb.
          tristate "KGDB: use kgdb over the serial console"
          select CONSOLE_POLL
          select MAGIC_SYSRQ
          default y
            Share a serial console with kgdb. Sysrq-g must be used
            to break in initially.

Code Optimization

Code optimization will harm the debugger stepping in a high level language (C in this case), because the high level statements necessarily map to exact one sequence of assembler statements. This results in jumping around with the stepping cursor in a debug session when sequential instruction processing is expected.

The only way to overcome this issue is to compile the code with -O0.

However, it is not possible to compile the whole kernel with -O0 since some few special code sequences can only be compiled with optimization.

Further, it is neither necessary nor a good idea to compile the whole kernel with -O0 for two reasons:

- In practice only a driver or e special subsystem contains the buggy code that has to be stepped through exactly - The kernel may behave different from the productive -O2 or -Os compilation that could lead to wrong conclusions

The kernel is compiled with '-Os' instead of '-O2' if CONFIG_CC_OPTIMIZE_FOR_SIZE is enabled. One may disable this option to have a weaker optimization but it's a better practice to add the following line to the Makefile of the driver/subsystem of interest:

ccflags-y := -O0

This will add -O0 after -O2 respectively -Os set generally by the kernel.

As stated by the GCC documentation:

The last option is effective:

If you use multiple -O options, with or without level numbers, the last such option is the one that is effective.

Additional debug info

It is possible to provide more debug info for gdb by using the -ggdb<LEVEL> option instead of -g e.g. including macro definitions.

Just change the top level Makefile of the kernel before compiling as follows:

KBUILD_AFLAGS	+= -gdwarf-2

Setup runtime for kgdb usage

  • Boot into Linux
  • Enable inetd so you can telnet into hardware
  • On the target, switch console tty device for use with kgdb
echo ttyS0 > /sys/module/kgdboc/parameters/kgdboc
echo g > /proc/sysrq-trigger

The SysRq command is also useful in order to stop kernel execution, sometimes it is normal to use at gdb the continue commands and the kernel will continue running without finding a break point, in order to stop kernel execution when desired send the SysRq command and kernel will stop at the gdb console:

echo g > /proc/sysrq-trigger

  • On the host, exit terminal emulator (picocom) and run gdb
cd $DEVDIR/kernel/linux-*
arm-linux-gnueabi-gdb -tui       # or maybe arm-none-linux-gnueabi-gdb -tui
file vmlinux
set remotebaud 115200
target remote /dev/ttyUSB3
break uart_register_driver

In telnet window, cause the breakpoint to be tripped

KDbg as GUI front end

NOTE: KDbg has an issue on Ubuntu 14.04.

Problem can be solved as described here

by removing the image file:

 sudo rm /usr/share/kde4/apps/kdbg/icons/hicolor/22x22/actions/pulse.mng

For a GUI front end KDbg vcan be used.

Installation in Ubuntu:

sudo apt-get install kdbg

To use the cross debugger instead of the host's gdb the global settings must be modified.

Start the front end:


Change the global settings:

Settings -> Global Options...

The Global Options dialog appears.

Setup How to invoke GDB: as follows if e.g. the Code Sourcery arm-2009q1 under /opt/codesourcery is used as toolchain:

/opt/codesourcery/arm-2009q1/bin/arm-none-linux-gnueabi-gdb --fullname --nx

For a different toolchain modify the path and the toolchain prefix acordingly.

Leave kdbg.

Now, if kgdb is started and waiting for a connection, you can connect with the KDbg front end as follows:

kdbg -r /dev/ttyUSB0 kernel/linux-

Adjust the kernel's vmlinux path and the serial port according to your configuration.

After about 5 seconds the source file kernel/kgdb.c should open stopping the execution at:

void kgdb_breakpoint(void)

Loadable modules and kgdb

Load the module and learn the load address.

modprobe sc16is7

You should get output something like

sc16is7 9793 0 - Live 0xbf000000

where 0xbf000000 is the module starting virtual address.

Tell the gdb about the module

add-symbol-file drivers/serial/sc16is7.o 0xbf000000


  • If you want to interrupt a running kernel and you get back to the gdb prompt
echo g > /proc/sysrq-trigger
  • Don't try to debug parts of the kernel that deal with scheduling/execution control. Don't even try to step over them with the next function. Instead, set a breakpoint after the function call to get gdb back in control. Some of the scheduling/execution control functions to watch out for include:

As you are stepping though code and approach one of these functions, such as:

   │199     int tty_buffer_request_room(struct tty_struct *tty, size_t size)
   │200     {
   │201             struct tty_buffer *b, *n;
   │202             int left;
   │203             unsigned long flags;
  >│205             spin_lock_irqsave(&tty->buf.lock, flags);
   │207             /* OPTIMISATION: We could keep a per tty "zero" sized buffer 
   │208                remove this conditional if its worth it. This would be
   │209                to the callers */
   │210             if ((b = tty->buf.tail) != NULL)
(gdb) s
tty_buffer_request_room (tty=0xc21b1000, size=1) at drivers/char/tty_buffer.c:200

You can skip the function by setting a temporary breakpoing on a line of code after the function call, such as:

(gdb) tbreak 210
Breakpoint 2 at 0xc018e548: file drivers/char/tty_buffer.c, line 210.
(gdb) continue
tty_buffer_request_room (tty=0xc21b1000, size=1) at drivers/char/tty_buffer.c:210

Then you can continue using gdb normally.