magic sysrq: a linux system debugging technique

Magic syrq is a feature in linux that helps users to debug kernel when other means of debugging fail for some reason.

Internals

In various kernel drivers - keyboard, serial, kgdb etc - the sysrq sequences are detected & handed to sysrq driver for special handling.  Unless the system in a state where even interrupt handlers are locked up, sysrq can be of rescue when other means of debugging the linux system fail for some reason.

How to enable sysrq

• Re-compile the kernel with CONFIG_MAGIC_SYSRQ macro enabled. We can find out whether magic sysrq feature is already compiled in by grep-ing .config file: 

root@babu-VirtualBox:/usr/lib# grep CONFIG_MAGIC_SYSRQ /boot/config-3.13.5

CONFIG_MAGIC_SYSRQ=y

CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE=0x1

root@babu-VirtualBox:/usr/lib# 

• To enable/disable at boot time, edit boot arguments to add "sysrq=1" or "sysrq=0" correspondingly.

• To enable/disable sysrq feature via keyboard commands at run time: 

echo "1" > /proc/sys/kernel/sysrq

OR

echo "0" > /proc/sys/kernel/sysrq

Without user supplying this configuration, this config is determined by CONFIG_MAGIC_SYSRQ_DEFAULT_ENABLE macro.

Also, these commands only enable/disable keyboard based sysrq. Other means of sysrq - via serial console/procfs - are not impacted by this config flag.

How to use syrq

Windows: Alt-<Syrq Key>-<character>. The character would be same as that echoed in the procfs commands listed below.
Mac: use procfs interface. Don't know the equivalent sequence of Alt-<SysRq>

Reboot the system (useful when system is hung):

echo b > /proc/sysrq-trigger

crash the system (used this so far to test some crash-triggered features):

echo c > /proc/sysrq-trigger

Show locks held up:

echo d > /proc/sysrq-trigger

Give control to debugger:

echo g > /proc/sysrq-trigger

Show backtrace of all cpus:

echo l > /proc/sysrq-trigger

Dump info about current tasks:

echo t > /proc/sysrq-trigger

Dump info about blocked tasks:

echo w > /proc/sysrq-trigger

There are many other keyboard sequences supported by sysrq, but the above are the ones I found significantly useful so far.

Note: For sysrq info to be printed on console, change console log level - "dmesg -n 5". If console log level is low, sysrq info can be found in "dmesg" output rather than on console.

Using sysrq via terminal server

TBD

References:
https://www.kernel.org/doc/Documentation/sysrq.txt