If I find a declining LowFree values in /proc/meminfo, then it is the kernel that is running low on memory. Then, I use the following steps to figure out which kernel component/module is causing the memory hog.
Slub Allocator Debug info:
To check whether any of the slab caches are consuming excessive memory, I run slabtop :
Android has another interesting tool - showslab - which can sort slab info based on various options such as: cache size, active obects etc.. In my VM, I too ported this from Android OS to linux x86. For those interested in using showslab on linux, here is my port: https://github.com/babuneelam/showslab_linux_x86_port. Original android source is available at https://android.googlesource.com/platform/system/extras/+/master/showslab/.
root@babu-VirtualBox:~/android_tools/bin#
./showslab -h
usage:
./showslab [options]
options:
-s
S specify sort criteria S
-h
display this help
Valid
sort criteria:
a:
number of Active objects
c:
Cache size
l:
number of sLabs
n:
Name
o:
number of Objects
p:
objects Per slab
s:
object Size
u:
cache Utilization
root@babu-VirtualBox:~/android_tools/bin#
root@babu-VirtualBox:~/android_tools/bin#
./showslab -s c
Active
/ Total Objects (% used) : 182094 / 185964 (97.9%)
Active
/ Total Slabs (% used) : 4651 / 4651 (100.0%)
Active
/ Total Caches (% used) : 66 / 97 (68.0%)
Active
/ Total Size (% used) : 26742.36K / 27477.89K (97.3%)
Min
/ Avg / Max Object Size : 0.01K / 0.15K / 8.00K
OBJS
|
ACTIVE
|
USE
|
OBJ SIZE
|
SLABS
|
OBJ/SLAB
|
CACHE SIZE
|
NAME
|
7008
|
7008
|
100%
|
0.63K
|
584
|
12
|
4672K
|
ext4_inode_cache
|
27296
|
26576
|
97%
|
0.12K
|
853
|
32
|
3412K
|
dentry
|
8602
|
8602
|
100%
|
0.34K
|
374
|
23
|
2992K
|
inode_cache
|
18186
|
17629
|
96%
|
0.09K
|
433
|
42
|
1732K
|
kmalloc-96
|
7938
|
7269
|
91%
|
0.19K
|
378
|
21
|
1512K
|
kmalloc-192
|
423
|
381
|
90%
|
3.22K
|
47
|
9
|
1504K
|
task_struct
|
26134
|
26134
|
100%
|
0.05K
|
358
|
73
|
1432K
|
buffer_head
|
3465
|
3465
|
100%
|
0.37K
|
165
|
21
|
1320K
|
proc_inode_cache
|
20032
|
20032
|
100%
|
0.06K
|
313
|
64
|
1252K
|
kmalloc-64
|
3497
|
3497
|
100%
|
0.30K
|
269
|
13
|
1076K
|
radix_tree_node
|
104
|
100
|
96%
|
8.00K
|
26
|
4
|
832K
|
kmalloc-8192
|
176
|
133
|
75%
|
4.00K
|
22
|
8
|
704K
|
kmalloc-4096
|
20224
|
19089
|
94%
|
0.03K
|
158
|
128
|
632K
|
kmalloc-32
|
1264
|
1200
|
94%
|
0.50K
|
79
|
16
|
632K
|
kmalloc-512
|
560
|
560
|
100%
|
1.00K
|
35
|
16
|
560K
|
kmalloc-1024
|
root@babu-VirtualBox:~/android_tools/bin#
root@babu-VirtualBox:~# cat /proc/slabinfo
#name
|
<active_objs>
|
<num_objs>
|
<objsize>
|
<objperslab>
|
<pagesperslab>
|
:tunables
|
<limit>
|
<batchcount>
|
<sharedfactor>
|
:slabdata
|
<active_slabs>
|
<num_slabs>
|
<sharedavail>
|
.
.
.
|
|||||||||||||
UDP
|
24
|
24
|
640
|
12
|
2
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
2
|
2
|
0
|
.
.
.
|
|||||||||||||
kmalloc-4096
|
133
|
144
|
4096
|
8
|
8
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
18
|
18
|
0
|
kmalloc-2048
|
96
|
96
|
2048
|
16
|
8
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
6
|
6
|
0
|
kmalloc-1024
|
480
|
480
|
1024
|
16
|
4
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
30
|
30
|
0
|
kmalloc-512
|
1248
|
1296
|
512
|
16
|
2
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
81
|
81
|
0
|
kmalloc-256
|
348
|
416
|
256
|
16
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
26
|
26
|
0
|
kmalloc-192
|
7299
|
7455
|
192
|
21
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
355
|
355
|
0
|
kmalloc-128
|
1176
|
1472
|
128
|
32
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
46
|
46
|
0
|
kmalloc-96
|
18765
|
19026
|
96
|
42
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
453
|
453
|
0
|
kmalloc-64
|
18624
|
18624
|
64
|
64
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
291
|
291
|
0
|
kmalloc-32
|
18577
|
19456
|
32
|
128
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
152
|
152
|
0
|
kmalloc-16
|
3072
|
3072
|
16
|
256
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
12
|
12
|
0
|
kmalloc-8
|
11264
|
11264
|
8
|
512
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
22
|
22
|
0
|
kmem_cache_node
|
256
|
256
|
32
|
128
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
2
|
2
|
0
|
kmem_cache
|
128
|
128
|
128
|
32
|
1
|
:tunables
|
0
|
0
|
0
|
:slabdata
|
4
|
4
|
0
|
root@babu-VirtualBox:~#
If the kernel component/module were using allocating/freeing using slab allocator, then the above tools help to corner down the memory hog to component/module level.
However, a lot of code in embedded systems has already been written using kmalloc/kfree. Because kmalloc/kfree now operate on top of slab allocator interface, slabtop does help to determine whether a kernel memory hog is creeping in even for such legacy kernel components/modules. However, it can't help corner down the source of memory hog/leak to component/module level as kmalloc fetches the memory from shared slab caches such as kmalloc-1024 rather than from a dedicated/named slab cache.
kmalloc Debug Info:
I haven't seen any lightweight tools that provide debug/stats for kmalloc/kfree.
At Brocade, I developed a tool called MLT (Memory Leak Tracking) that provides a list of allocating code stack traces sorted by outstanding kmallocs. This is simply a Kernel Memory Accounting tool misnamed as leak tracking tool, though memory accounting helps to track down even unchecked memory leaks. I achieved this by replacing kmalloc/kfree with wrapper functions & accounting the allocs/frees in these functions. MLT helped me troubleshoot significant number of kernel memory leaks to the level of stack trace instantly. I mostly didn't even have to look at /proc/slabinfo or /proc/buddyinfo. MLT can be deployed even in customer environments & hence could be very useful in troubleshooting customer issues.
As Brocade released the source code of MLT due to open source obligations of Linux, I could copy the code from Brocade's website & publish at https://github.com/babuneelam/mlt_brocade. Please feel free to try it out.
On somewhat similar lines of MLT concept, I found the following patches in linux:
http://lwn.net/Articles/124374/
https://lkml.org/lkml/2005/2/20/73
I wish this patch were part of linux kernel by default, but it is not as of now.
No comments:
Post a Comment