Wednesday, March 26, 2014

Monitor stack limits and usage at runtime in linux


User Space Process Stack

The stack size of a process is limited by two configuration values: soft limit and hard limit. At any time, a process can't exceed usage beyond soft limit. Hard limit is a ceiling for soft limit. An under privileged process can only lower it's hard limit, but can't increase. A privileged process can increase/decrease hard limit. Refer to setrlimit man page to understand how to change hard/soft limit during the execution of a process.

How to find system wide stack size limits applicable by default to a process?
root@babu-VirtualBox:~# ulimit -a | grep stack
stack size              (kbytes, -s) 8192
root@babu-VirtualBox:~# ulimit -s
8192
root@babu-VirtualBox:~# 

How to find the stack size limits of a particular process?
root@babu-VirtualBox:~# cat /proc/2338/limits 
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            8388608              unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             unlimited            unlimited            processes 
Max open files            1024                 4096                 files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       6149                 6149                 signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        
root@babu-VirtualBox:~#

I wrote a script (located at https://github.com/babuneelam/stk_corruption/blob/master/process_stk_limits.sh) that helps to list the stack size limits of all user-space processes currently running in the system.
root@babu-VirtualBox:~/tools/stk_corruption# ./process_stk_limits.sh 
Current soft/hard limits of various processing running in the system now:
PID     Soft Limit       Hard Limit     Units   Proc_name
973     8388608        unlimited        bytes  /sbin/getty-838400tty6
968     8388608        unlimited        bytes  /sbin/getty-838400tty3
967     8388608        unlimited        bytes  /sbin/getty-838400tty2
959     8388608        unlimited        bytes  /sbin/getty-838400tty5
953     8388608        unlimited        bytes  /sbin/getty-838400tty4
719     8388608        unlimited        bytes  /usr/sbin/dnsmasq....
7088   8388608        unlimited        bytes  /bin/bash./process_stk_limits.sh
.
.
.
.
root@babu-VirtualBox:~/tools/stk_corruption# 

How to find current allocated stack size/usage of a process?
root@babu-VirtualBox:~/tools/stk_corruption# cat /proc/2338/status | grep -i vmstk
VmStk:      136 kB
root@babu-VirtualBox:~/tools/stk_corruption# 

I wrote a script (located at https://github.com/babuneelam/stk_corruption/blob/master/process_stk_usage.sh) that helps to list the stack size usage of all user-space processes currently running in the system.
root@babu-VirtualBox:~/tools/stk_corruption# ./process_stk_usage.sh 
Current soft/hard limits of various processing running in the system now:
PID     Current_Usage         Proc_name
392     156kb rsyslogd-c5  
1043   148kb /usr/bin/X-core:0-auth/var/run/lightdm/root/:0-nolistentcpvt7-novtswitch   
973     136kb /sbin/getty-838400tty6      
968     136kb /sbin/getty-838400tty3      
967     136kb /sbin/getty-838400tty2      
959     136kb /sbin/getty-838400tty5      
953     136kb /sbin/getty-838400tty4      
8714   136kb /bin/bash./process_stk_usage.sh  
6906   136kb /usr/bin/atop-a-w/var/log/atop/atop_20140327600      
679     136kb /usr/sbin/cups-browsed      
666     136kb /usr/lib/policykit-1/polkitd--no-debug      
656     136kb NetworkManager   
596     136kb /usr/sbin/modem-manager  
.
.
.
.
root@babu-VirtualBox:~/tools/stk_corruption# 

 How to find the current stack trace of a uspace process?
root@babu-VirtualBox:~/tools/stk_corruption# cat /proc/2469/stack
[<c1bb0da6>] unix_stream_recvmsg+0x416/0xce0
[<c1a8151b>] sock_aio_read+0x17b/0x200
[<c12f89c1>] do_sync_read+0x81/0xe0
[<c12f9ddf>] vfs_read+0x1ff/0x260
[<c12fa0e5>] SyS_read+0x85/0x110
[<c1c77acd>] sysenter_do_call+0x12/0x28
[<ffffffff>] 0xffffffff
root@babu-VirtualBox:~/tools/stk_corruption#

Kernel Space Thread Stack

The size of a kernel side stack for either user space thread or a stack for kernel thread is fixed at compile time & is determined by THREAD_SIZE macro. This value is different for different architectures. Historically, this was 4K, but now this is 8K (http://lxr.linux.no/linux+v3.13.5/arch/x86/include/asm/page_32_types.h#L21). However, I have worked on embedded system which increased the kernel stack size to 16K. I am not aware of the need to have such huge stack - TBD.

How to find stack size limit of a kernel thread at run time?
I don't know of any command line tool that outputs kernel's stack size in linux. Unfortunately, the command /proc/<pid>/status, when used on kernel threads, is not showing stack usage info (as it shows for user space processes).

How to find the run-time stack usage of different kernel threads?
This information is not available by default in linux kernel. The flag CONFIG_DEBUG_STACK_USAGE should be set to "y" or enabled. After enabling, recompile the kernel & reload the image. Then, below are the steps to find the run time stack usage of different kernel threads:
root@babu-VirtualBox:~/tools/stk_corruption# dmesg -c
root@babu-VirtualBox:~/tools/stk_corruption# echo t > /proc/sysrq-trigger
root@babu-VirtualBox:~/tools/stk_corruption# dmesg > dmesg_stk_usage
root@babu-VirtualBox:~/tools/stk_corruption# cat dmesg_stk_usage | grep " S " | cut -d ' ' -f 2-
                               PC               Stack  PID    Parent
dconf worker
S
f73de43c  
6440
2342
1792
0x00000000
gdbus        
S
f73e3540
6440
2343
1792
0x00000000
update-notifier 
S
f73de43c
6072
2423
1792
0x00000000
dconf worker
S
f73ec43c
6224
2426
1792
0x00000000
gdbus           
S
f73f1540
6200
2427
1792
0x00000000
gmain        
S
f73de43c  
6196
2428
1792
0x00000000
deja-dup-monito
S
e401e208
6196
2474
1792
0x00000000
dconf worker
S
f73ec43c  
6212
2477
1792
0x00000000
gdbus      
S
f6c01b00
6216
2478
1792
0x00000000
kworker/u4:2
S
eddb6480 
5652
3648
2
0x00000000
kworker/1:2
S
00000001
7260
3822
2
0x00000000
kworker/0:0
S
00000000
7260
5170
2
0x00000000
update-manager
S
f73de43c  
6052
5827
1601
0x00000000
dconf worker
S
f73ec43c
6224
5828
1601
0x00000000
gdbus
S
f73f1540  
6224
5829
1601
0x00000000
kworker/u5:2
S
f6467ed4
7232
5913
2
0x00000000
dhclient
S
f73ec43c  
6264
6038
656
0x00000000
kworker/u4:0
S
 00000001
7260
6810
2
0x00000000
atop 
S
00000008
6560
6906
1
0x00000000
oneconf-service
S
f73ec43c  
6216
9798
1601
0x00000000
gmain        
S
00000000
6440
9799
1601
0x00000000
dconf worker
S
f73de43c  
6224
9800
1601
0x00000000
gdbus
S
f73e3540  
6440
9802
1601
0x00000000 
cupsd          
S
f73de43c  
6364
9932
1
0x00000000
dbus 
S
f5e7fe08  
6368
9935
9932
0x00000000
root@babu-VirtualBox:~/tools/stk_corruption# 

How to find the current stack trace of a kernel thread?
root@babu-VirtualBox:~/tools/stk_corruption# cat /proc/132/stack
[<c14a0707>] kjournald2+0x2f7/0x490
[<c10e6985>] kthread+0x115/0x160
[<c1c77a37>] ret_from_kernel_thread+0x1b/0x28
[<ffffffff>] 0xffffffff
root@babu-VirtualBox:~/tools/stk_corruption#

How to find know which kernel threads are building up towards stack overflow situation?

  • One way I can think of: Combine the techniques I wrote above. Have a scripting tool that logs stack usage & the current stack of all kernel threads at regular intervals to the disk. And another scripting tool that helps report this raw data in a intuitive manner. Such a tool would help to proactively look for potential kernel stack overflows building up in the system. However, such a tool can reduce system performance depending on how frequently the stack usage profile information is collected by the tool.


 IRQ Stack

TBD
  • CONFIG_DEBUG_STACKOVERFLOW - TBD. Prints message when free stack space drops below a certain limit.. kernel runs a little bit slower due to the proactive additional checks done to ensure that stack doesn't overflow, but catches most overflows. However, these checks are done only in IQR contexts, but not in kernel threads!!

References:
http://lxr.linux.no/linux+v3.13.5/Documentation/x86/x86_64/kernel-stacks

Posted by at
Labels: , , , , , ,

No comments:

Post a Comment

Subscribe to: Post Comments (Atom)
UA-48797665-1