Kernel Memory Leak Detection: kmemleak overview

I recently experimented a linux kernel based memory leak detection tool - kmemleak. In this post, I will cover kmemleak basics.

What does kmemleak do? 
Detect memory leaks in linux kernel. Provides allocation stack trace & the pointer that is considered a memory leak.

Memory Leak Detection Methodology:
kmemleak tracks allocs/frees in linux kernel by tracking kmalloc/kfree, kmem_cache_alloc/kmem_cache_free and vmalloc/vfree. It keeps all outstanding allocated pointers, size allocated & the allocation stack trace in it's data structure. On a end user trigger or on periodic basis (configurable), it scans kernel data section, heap, bss and registers to check the number of references currently pointing to these outstanding allocated pointers or anywhere inside the memory chunks they are pointing to. At any time, if any of the pointers and the corresponding memory chunks are found to have no references, then kmemleak reports this pointer as a memory leak & also provides the allocation trace for further debugging.

One of the advantages of kmemleak is that developer can understand memory leaks in their code proactively rather than wait until this emory leak eventually causes an OOM.
 

Drawbacks:

• Requires Kernel re-compilation
• Performance intensive - because kmemleak scans entire kernel memory periodically!! but the impact is lessened because it has the configurable option of scanning only on user request. Even then, kmemleak can't be used in customer deployments. This can be useful in developer environments though.
• False negatives - because kmemleak scan even non-pointers in kernel memory.
• False positives - because some code derives a pointer from other pointers it may have without storing it in memory!!

 

End User How To:
 

• In linux, kmemleak is not enabled by default. To enable kmemleak, Linux kernel should be recompiled after enabling CONFIG_DEBUG_KMEMLEAK. Here are the steps I used to enable kmemleak in ubuntu 13.10 distribution:

              root@babu-VirtualBox:/usr/src/linux-3.11# make menuconfig

              This took me to the following screens successively:

              Here, I selected SLUB debugging, SLUB performance stats, Kernel memory leak detector:

              After this, the usual kernel compilation as I mentioned in my other 

              post: How to - Linux Kernel Compilation in Ubuntu VM/VirtualBox.

• After enabling kmemleak, it can also be disabled at boot-time by passing "kmemleak=off" on the kernel command line.

• Triggering a scan

If debugfs is not already mounted,
 root@babu-VirtualBox:~# mount -t debugfs nodev /sys/kernel/debug/

 root@babu-VirtualBox:~# echo scan > /sys/kernel/debug/kmemleak

• Checking for current memory leaks

                   root@babu-VirtualBox:~# cat /sys/kernel/debug/kmemleak 
                   unreferenced object 0xe0a7fcf0 (size 8):
                   comm "systemd-udevd", pid 307, jiffies 4294895134 (age 169.028s)
                   hex dump (first 8 bytes):
                       00 00 00 00 00 00 00 00                          ........
                   backtrace:
                       [<c161785c>] kmemleak_alloc+0x3c/0xa0
                       [<c115709b>] __kmalloc+0x12b/0x1c0
                       [<f8485d0d>] exit_nls_utf8+0xc9b/0xf8e [nls_utf8]
                       [<c10020ca>] do_one_initcall+0xca/0x190
                       [<c10b4524>] load_module+0x1ae4/0x2460
                       [<c10b4f2f>] SyS_init_module+0x8f/0xf0
                       [<c162c54d>] sysenter_do_call+0x12/0x28
                       [<ffffffff>] 0xffffffff
                   .
                   .

                   .

                   root@babu-VirtualBox:~#

•  Clearing currently reported memoryleaks

root@babu-VirtualBox:~# echo clear > /sys/kernel/debug/kmemleak

• Other Available Configuration Options 

                  Memory scanning parameters can be modified at run-time by writing to the

                  /sys/kernel/debug/kmemleak file. The following parameters are supported:

                                   off                     - disable kmemleak (irreversible) 
                                   stack=on           - enable the task stacks scanning (default) 
                                   stack=off          - disable the tasks stacks scanning

                                   scan=on            - start the automatic memory scanning thread (default) 
                                   scan=off           - stop the automatic memory scanning thread 
                                   scan=<secs>     - set the automatic memory scanning period in seconds 
                                                             (default 600, 0 to stop the automatic scanning) 
                                   dump=<addr>  - dump information about the object found at <addr>