In my earlier post - Linux kernel debuggers: Overview - I covered the basic features of kgdb. In this post, I will provide steps to use kgdb.
kgdb usage - HowTo
Here are the steps that worked for me to debug a VitualBox Ubuntu VM. I used two VMs: VMDebugged - the VM that is being debugged & VMDebugger - The VM that helps debug VMDebugged.
- Recompile the kernel in VMDebugged with kgdb options enabled. We can find out whether its already compiled in by grep-ing for KGDB compile macros:
VMDebugged
root@babu-VirtualBox:~# grep KGDB
/boot/config-3.13.5
CONFIG_SERIAL_KGDB_NMI=y
CONFIG_HAVE_ARCH_KGDB=y
CONFIG_KGDB=y
CONFIG_KGDB_SERIAL_CONSOLE=y
#
CONFIG_KGDB_TESTS is not set
CONFIG_KGDB_LOW_LEVEL_TRAP=y
CONFIG_KGDB_KDB=y
root@babu-VirtualBox:~#
- Copy the vmlinux file from VMDebugged to VMDebugger.
- To establish debugging using kgdb, we need to configure console redirection from VMDebugged to VMDebugger - create host pipe based serial ports as described in my other blog post - http://techvolve.blogspot.com/2013/11/how-to-redirect-console-output-of-one.html.
- Boot VMDebugger. VMDebugger should be started before VMDebugged so that pipe is created by VMDebugger & it is attached to by VMDebugged.
- gdb the vmlinux file & start debugging over serial port. Here, the debugger just waits as the VMDebugged is still not up.
VMDebugger
root@babu-VirtualBox:/media/sf_Linux_Code# gdb
./vmlinux
GNU gdb (GDB) 7.6.1-ubuntu
Copyright (C) 2013 Free Software Foundation,
Inc.
License GPLv3+: GNU GPL version 3 or later
<http://gnu.org/licenses/gpl.html>
This is free software: you are free to change
and redistribute it.
There is NO WARRANTY, to the extent permitted by
law. Type "show copying"
and "show warranty" for details.
This GDB was configured as
"i686-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>...
Reading symbols from
/media/sf_Linux_Code/vmlinux...done.
(gdb) set remotebaud 115200
(gdb) target remote /dev/ttyS0 (or target remote <terminal_server_ip>:<port> if debugging via terminal server)
Remote debugging using
/dev/ttyS0
- Rather than edit boot options as mentioned in the above blog post, edit boot options of VMDebugged to enable kgdb, let debugging control via serial port, wait kernel initialization until requested by debugger in VMDebugger. To do so, add "kgdboc=ttyS0,115200 kgdbwait" to the boot options of VMDebugged.
- kgdboc stands for kgdb over console. This config parameter allows us to set the interface mechanism with kgdb/gdb: <serial_port>[,<baud_rate>]
- kgdbwait config parameter allows us to enable early debugging. If this is specified, the kernel waits as early as possible and waits until the front end requests to resume.
VMDebugged
- Boot VMDebugged.
- Now, VMDebugged stops before kernel initialization. And the debugger terminal in VMDebugged gets debugger control.
VMDebugged
VMDebugger
0xc11bca3f in arch_kgdb_breakpoint ()
at
/usr/src/linux-3.13.5/arch/x86/include/asm/kgdb.h:80
80 asm(" int $3");
(gdb)
(gdb)
- Then, just allow the booting process to continue in VMDebugger & let VMDebugged completely boot:
VMDebugger
(gdb) c
Continuing.
[New Thread 2327]
VMDebugged
- Even if kgdb is not enabled in boot parameters, kgdb can be enabled at run time in sysfs interface.
To enable kgdb at run-time:
root@babu-VirtualBox:~# cat /sys/module/kgdboc/parameters/kgdboc
ttyS0,115200
root@babu-VirtualBox:~#
To
disable kgdb at run-time:
root@babu-VirtualBox:~#
echo "" > /sys/module/kgdboc/parameters/kgdboc
root@babu-VirtualBox:~#
To check whether kgdb is enabled/disabled:
root@babu-VirtualBox:~# cat /sys/module/kgdboc/parameters/kgdboc
ttyS0,115200
root@babu-VirtualBox:~#
- Now, to bring the control back to debugger, type "echo g > /proc/sysrq-trigger" in VMDebugged.
VMDebugged
root@babu-VirtualBox:~# echo g > /proc/sysrq-trigger
VMDebugger
Program received signal SIGTRAP,
Trace/breakpoint trap.
[Switching to Thread 2327]
0xc11bca3f in arch_kgdb_breakpoint ()
at
/usr/src/linux-3.13.5/arch/x86/include/asm/kgdb.h:80
80 asm(" int $3");
(gdb) bt
#0 0xc11bca3f in arch_kgdb_breakpoint ()
at
/usr/src/linux-3.13.5/arch/x86/include/asm/kgdb.h:80
#1 kgdb_breakpoint () at
kernel/debug/debug_core.c:1041
#2 0xc11bcb09 in sysrq_handle_dbg
(key=<optimized out>)
at kernel/debug/debug_core.c:801
#3 0xc179b85f in __handle_sysrq (key=103,
check_mask=check_mask@entry=false)
at drivers/tty/sysrq.c:535
#4 0xc179ba9c in write_sysrq_trigger
(file=file@entry=0xf33e5e40,
buf=buf@entry=0x83e1c08
"g\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
ppos=ppos@entry=0xf4523f98)
at drivers/tty/sysrq.c:1083
#5 0xc139fc0e in proc_reg_write
(file=file@entry=0xf33e5e40,
buf=buf@entry=0x83e1c08
"g\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
ppos=ppos@entry=0xf4523f98) at fs/proc/inode.c:224
#6 0xc12f9989 in vfs_write
(file=file@entry=0xf33e5e40,
buf=buf@entry=0x83e1c08
"g\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
pos=pos@entry=0xf4523f98) at fs/read_write.c:473
#7 0xc12fa1f5 in SYSC_write (count=<optimized
out>,
buf=0x83e1c08 "g\nlgrind is
/usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n", '\337' <repeats 143
times>, <incomplete sequence \337>...,
---Type <return> to continue, or q
<return> to quit---
fd=<optimized out>) at
fs/read_write.c:522
#8 SyS_write (fd=1, buf=138288136,
count=2) at fs/read_write.c:514
#9 <signal handler called>
#10 0xb776d424 in ?? ()
#11 0x083e1c08 in ?? ()
Cannot access memory at address 0xfbad2a89
(gdb)
- For testing purposes, I then crash the VMDebugged. In case of crash, the debugger in VMDebugger automatically gets the control for further inspection.
VMDebugged
root@babu-VirtualBox:~# echo c > /proc/sysrq-trigger
VMDebugger
Program received signal SIGSEGV, Segmentation
fault.
sysrq_handle_crash (key=99) at
drivers/tty/sysrq.c:140
140 *killer = 1; <-- The line at which system crashed
(gdb) bt
(gdb) bt
#0 sysrq_handle_crash (key=99) at drivers/tty/sysrq.c:140 <-- stack trace of the crash
#1 0xc179b85f in __handle_sysrq (key=99,
check_mask=check_mask@entry=false)
at drivers/tty/sysrq.c:535
#2 0xc179ba9c in write_sysrq_trigger
(file=file@entry=0xf3a32240,
buf=buf@entry=0x83e1c08
"c\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
ppos=ppos@entry=0xf4523f98)
at drivers/tty/sysrq.c:1083
#3 0xc139fc0e in proc_reg_write
(file=file@entry=0xf3a32240,
buf=buf@entry=0x83e1c08
"c\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
ppos=ppos@entry=0xf4523f98) at fs/proc/inode.c:224
#4 0xc12f9989 in vfs_write
(file=file@entry=0xf3a32240,
buf=buf@entry=0x83e1c08
"c\nlgrind is /usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n",
'\337' <repeats 143 times>, <incomplete sequence \337>...,
count=count@entry=2,
pos=pos@entry=0xf4523f98) at fs/read_write.c:473
#5 0xc12fa1f5 in SYSC_write
(count=<optimized out>,
buf=0x83e1c08 "c\nlgrind is
/usr/bin/valgrind\ngrub-script-check\nbkdf2\ne\n", '\337' <repeats 143
times>, <incomplete sequence \337>...,
fd=<optimized out>) at
fs/read_write.c:522
#6 SyS_write (fd=1, buf=138288136,
count=2) at fs/read_write.c:514
#7 <signal handler called>
#8 0xb776d424 in ?? ()
---Type <return> to continue, or q
<return> to quit---q
Quit
(gdb)
Hope this helps.
References:
http://sourceware.org/gdb/current/onlinedocs/gdb/
No comments:
Post a Comment