If I find a declining HighFree values in /proc/meminfo, then it is mostly some user space application that is causing or memory hog. Then, I use the following steps to figure out the process & the code path inside the process that is causing the memory hog.
Step1: Identify the process that is consuming increasing levels of memory
procrank (if available), ps and top are the commands I use to identify the process(es) that are consuming huge amount of memory in the system.
Before I delve into the details of the above commands, here is some terminology related to these commands:
- VSS/VSZ (Virtual Set/memory Size): The amount of virtual memory allocated to the process. Physical memory pages may not have to be allocated to be counted in VSS/VSZ. Device mappings are currently excluded.
- RSS (Resident Set/Memory Size): The amount of physical memory - only no-swapped - currently allocated to the process. Device mappings are currently excluded. All shared memory is counted fully into each process' RSS. So, summing up RSS of all processes will overcount the amount of memory actually used.
- PSS (Proportional Set/Memory Size): The process' proportional share of RSS. The amount of unique memory mapped to the process + shared memory used by process divided by the number of processes sharing it (proportional).
- USS (unique Set/Memory Size): The process' private share of RSS. The amount of unique memory mapped to the process. This is the amount of memory that would be freed if the process were terminated now.
Basically, the goal would be to identify the process whose PSS/USS (or RSS if PSS/USS can't be obtained) is increasing continuously & hence causing memory hog.
procrank is available in Android OS. However, I have used this even in embedded systems which ported this tool from Android OS to linux. In my VM, I too ported this from Android OS to linux x86. For those interested in using procrank on linux, here is my port: https://github.com/babuneelam/procrank_linux_x86_port. Original android source is available at https://android.googlesource.com/platform/system/extras/+/master/procrank/.
root@babu-VirtualBox:~#
./procrank -h
Usage:
./procrank [ -W ] [ -v | -r | -p | -u | -s | -h ]
-v Sort by VSS.
-r Sort by RSS.
-p Sort by PSS.
-u Sort by USS.
-s Sort by swap.
(Default sort order is PSS.)
-R Reverse sort order (default is descending).
-c Only show cached (storage backed) pages
-C Only show non-cached (ram/swap backed) pages
-k Only show pages collapsed by KSM
-w Display statistics for working set only.
-W Reset working set of all processes.
-h Display this help screen.
root@babu-VirtualBox:~#
root@babu-VirtualBox:~# ./procrank
PID
|
Vss
|
Rss
|
Pss
|
Uss
|
cmdline
|
2061
|
437380K
|
168036K
|
154486K
|
151408K
|
compiz
|
987
|
188756K
|
46116K
|
39385K
|
34960K
|
/usr/bin/X
|
2145
|
127396K
|
30748K
|
25520K
|
24744K
|
/usr/lib/evolution/evolution-calendar-factory
|
2085
|
208080K
|
23880K
|
10937K
|
8256K
|
nautilus
|
2263
|
97228K
|
16104K
|
9831K
|
8720K
|
/usr/bin/unity-scope-loader
|
2320
|
137436K
|
17248K
|
7305K
|
6256K
|
gnome-terminal
|
1823
|
144780K
|
14616K
|
6234K
|
5472K
|
/usr/lib/gnome-settings-daemon/gnome-settings-daemon
|
.
.
.
.
---------- ------------ ----------
405365K 369332K TOTAL
RAM:
1026276K total, 283124K free, 96632K buffers, 280652K cached, 6276K shmem,
28036K slab
root@babu-VirtualBox:~#
root@babu-VirtualBox:~# ps -eo
user,pid,rss,vsz,sz,stat,cmd --sort -rss
USER
|
PID
|
RSS
|
VSZ
|
SZ
|
STAT
|
CMD
|
babu
|
1999
|
155356
|
408284
|
102071
|
Sl
|
compiz
|
babu
|
2431
|
61240
|
191084
|
47771
|
SNl
|
/usr/bin/python3 /usr/bin/update-manager --no-update
--no-focus-on-map
|
root
|
1024
|
34856
|
149804
|
37451
|
Ssl+
|
/usr/bin/X -core :0 -auth /var/run/lightdm/root/:0
-nolisten tcp vt7 -novtswitch
|
babu
|
2106
|
30972
|
128896
|
32224
|
Sl
|
/usr/lib/evolution/evolution-calendar-factory
|
babu
|
2052
|
23024
|
199860
|
49965
|
Sl
|
nautilus -n
|
babu
|
2296
|
17272
|
138364
|
34591
|
Sl
|
gnome-terminal
|
babu
|
1770
|
16200
|
135480
|
33870
|
Ssl
|
/usr/lib/unity/unity-panel-service
|
babu
|
2221
|
15756
|
97312
|
24328
|
Sl
|
/usr/bin/unity-scope-loader applications/applications.scope
applications/scopes.scope commands.scope
|
babu
|
2025
|
15480
|
189888
|
47472
|
Sl
|
nm-applet
|
babu
|
1740
|
14872
|
219780
|
54945
|
Ssl
|
/usr/lib/gnome-settings-daemon/gnome-settings-daemon
|
babu
|
1989
|
13916
|
107932
|
26983
|
Sl
|
/usr/lib/i386-linux-gnu/notify-osd
|
.
.
.
root@babu-VirtualBox:~#
I already explained RSS & VSZ above. For other fields of ps output, here is some help directly copied from ps man page:
SZ size in physical pages of the core image of the process.
This includes text, data, and stack space. Device mappings
are currently excluded; this is subject to change. See vsz
and rss.
STAT multi-character process state. See section PROCESS STATE
CODES for the different values meaning. See also s and
state if you just want the first character displayed.
Here are the different values that the s, stat and state output specifiers (header "STAT" or "S") will
display to describe the state of a process.
D Uninterruptible sleep (usually IO)
S Interruptible sleep (waiting for an event to complete)
T Stopped, either by a job control signal or because it is being traced.
W paging (not valid since the 2.6.xx kernel)
X dead (should never be seen)
Z Defunct ("zombie") process, terminated but not reaped by its parent.
Typically, procrank/ps would suffice to get the top memory consuming process. However, 'top' is also another popular command that can help get the same info. After entering top command, pressing 'M' would sort the top output by memory consumed:
Following are the memory related fields to focus in 'top' output:
- %MEM -- Memory Usage (RES): A task's currently used share of available physical memory.
- VIRT -- Virtual Memory Size (KiB): The total amount of virtual memory used by the task. It includes all code, data and shared libraries plus pages that have been swapped out and pages that have been mapped but not used.
- VIRT = SWAP+RES
- RES -- Resident Memory Size (KiB): The non-swapped physical memory a task has used.
- RES = CODE+DATA
- CODE -- Code Size (KiB): The amount of physical memory devoted to executable code, also known as the 'text resident set' size or TRS.
- DATA -- Data + Stack Size (KiB): The amount of physical memory devoted to other than executable code, also known as the 'data resident set' size or DRS.
- SHR -- Shared Memory Size (KiB): The amount of shared memory available to a task, not all of which is typically resident. It simply reflects memory that could be potentially shared with other processes.
- SWAP -- Swapped Size (KiB): The non-resident portion of a task's address space.
I examine which section/map of the process is growing to determine if there is a memory hog. Sometimes, the process will need more memory for it's functionality & the memory growth might just stabilize after sometime.
Android has procmem utility that shows all sections/maps of a process sorted by PSS size. In my VM, I too ported this from Android OS to linux x86. For those interested in using showmem on linux, here is my port: https://github.com/babuneelam/procmem_linux_x86_port. Original android source is available at https://android.googlesource.com/platform/system/extras/+/master/procmem/.
root@babu-VirtualBox:~/android_tools/bin#
./procmem
Usage:
./procmem [ -w | -W ] [ -p | -m ] [ -h ] pid
-w Displays statistics for the working set only.
-W Resets the working set of the process.
-p Sort by PSS.
-m Sort by mapping order (as read from /proc).
-h Hide maps with no RSS.
root@babu-VirtualBox:~/android_tools/bin#
./procmem -p 2977
|
Vss
|
Rss
|
Pss
|
Uss
|
ShCl
|
ShDi
|
PrCl
|
PrDi
|
Name
|
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
|
|
10040K
|
9952K
|
9952K
|
9952K
|
0K
|
0K
|
9952K
|
0K
|
[heap]
|
|
884K
|
680K
|
370K
|
60K
|
620K
|
0K
|
60K
|
0K
|
/bin/bash
|
|
132K
|
28K
|
28K
|
28K
|
0K
|
0K
|
28K
|
0K
|
[stack]
|
|
120K
|
92K
|
23K
|
0K
|
92K
|
0K
|
0K
|
0K
|
/lib/i386-linux-gnu/libtinfo.so.5.9
|
|
1720K
|
716K
|
23K
|
0K
|
716K
|
0K
|
0K
|
0K
|
/lib/i386-linux-gnu/libc-2.17.so
|
|
20K
|
20K
|
20K
|
20K
|
0K
|
0K
|
20K
|
0K
|
/bin/bash
|
|
.
|
|
|
|
|
|
|
|
|
|
.
|
|
|
|
|
|
|
|
|
|
.
|
|
|
|
|
|
|
|
|
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
-------
|
|
|
15472K
|
11892K
|
10554K
|
10184K
|
1708K
|
0K
|
10184K
|
0K
|
TOTAL
|
root@babu-VirtualBox:~/android_tools/bin#
A few more related sections to check in procmem:
- [heap] section - heap section. A growth in this section is likely an indication of a memory hog.
- [heap] section - heap section. A growth in this section is likely an indication of a memory hog.
- [anon] section - anonymous mappings done by malloc() in glibc. Growth of this section indicates memory hog.
- [vdso] section - kernel syscall helper. This section should shouldn't grow.
- [stack] - This section shouldn't grow. If this grows, the application has buggy code.
- [stack] - This section shouldn't grow. If this grows, the application has buggy code.
Without procmem port, smaps can be used. smaps subsystem is a successor of /proc/<pid>/maps. I check whether PSS of heap/anon sections is growing in /proc/<pid>/smaps. smaps file is typically a very big file & hence reading it is very difficult.
root@babu-VirtualBox:~# cat /proc/2304/smaps | grep -A 15 heap
0a01c000-0a111000 rw-p 00000000 00:00 0 [heap]
Size:
|
980 kB
|
Rss:
|
980 kB
|
Pss:
|
980 kB
|
Shared_Clean:
|
0 kB
|
Shared_Dirty:
|
0 kB
|
Private_Clean:
|
0 kB
|
Private_Dirty:
|
980 kB
|
Referenced:
|
980 kB
|
Anonymous:
|
980 kB
|
AnonHugePages:
|
0 kB
|
Swap:
|
0 kB
|
KernelPageSize:
|
4 kB
|
MMUPageSize:
|
4 kB
|
Locked:
|
0 kB
|
VmFlags: rd wr mr mw me ac
root@babu-VirtualBox:~# Once the top memory consuming process is identified, the next step is find the code path within the process that is allocating this memory.
I am yet to explore a open source memory profiler.
I came across a method of malloc/free accounting dynamic library that can be enabled/disabled run time without needing compile time instrumentation. Such libraries tap into malloc/free wrappers of glibc to do the memory accounting. By providing enable/disable configuration knob and disabling it by default, such library will not degrade application performance. This can be useful to troubleshoot memory hogs in developer environments. However, by having this knob disabled by default, the usefulness of such library would be limited in troubleshooting customer issues without reproducing the customer defect after enabling this knob.
No comments:
Post a Comment