How to - gcov/lcov for linux user space process

To test & explore gcov/lcov functionality for user space applications, here are the steps I used:

• I wanted to get coverage for a few android tools that I ported to linux a while ago. Here is the code: https://github.com/babuneelam/gcov_uspace_tests. Here is the structure of my applications' source code:

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  latencytop  Makefile  sane_schedstat  showmap

./bin:

./latencytop:

DumpGcovData  latencytop.c  Makefile

./sane_schedstat:

Makefile  sane_schedstat.c

./showmap:

Makefile  showmap.c

root@babu-VirtualBox:~/gcov_tests# 

• To instrument the code with gcov, I enabled gcov compilation flags in two applications latencytop & sane_schedstat. I also turned off any gcc optimization by removing "O" flags :

root@babu-VirtualBox:~/gcov_tests# grep -irne "-fprofile-arcs" *

latencytop/Makefile:6:CFLAGS       = -fPIC -pedantic -fprofile-arcs -ftest-coverage -march=native -ggdb3

sane_schedstat/Makefile:6:CFLAGS       = -fPIC -pedantic -fprofile-arcs -ftest-coverage -march=native -ggdb3

root@babu-VirtualBox:~/gcov_tests# 

• I compiled all the applications from top directory:

root@babu-VirtualBox:~/gcov_tests# make

• After compilation, I see that gcc generated .gcov files in the two applications instrumented with gcov flags:

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  latencytop  Makefile  sane_schedstat  showmap

./bin:

latencytop  sane_schedstat  showmap

./latencytop:

latencytop.c  latencytop.gcno  latencytop.o  Makefile

./sane_schedstat:

Makefile  sane_schedstat.c  sane_schedstat.gcno  sane_schedstat.o

./showmap:

Makefile  showmap.c  showmap.o

root@babu-VirtualBox:~/gcov_tests# 

• I then executed latencytop & sane_schedstat binaries in two different terminals. Both these applications run in a loop. So, these commands kept looping in those terminals.

          
               

• I then dumped gcov run time date using a custom script I wrote - https://github.com/babuneelam/gcov_scripts/blob/master/dump_gcov. This script basically attaches to the process through gdb & flushes gcov run time data & then detaches from gdb.

root@babu-VirtualBox:~/gcov_tests# ./dump_gcov latencytop  sane_schedstat

Application: latencytop 
Dumping gcov data...
Gcov data dump complete.

Application: sane_schedstat 
Dumping gcov data...
Gcov data dump complete.
  
root@babu-VirtualBox:~/gcov_tests# 

• After this, I see that dump_gcov script generated .gcda files in the two applications instrumented with gcov flags.

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  dump_gcov  latencytop  Makefile  sane_schedstat  showmap

./bin:

latencytop  sane_schedstat  showmap

./latencytop:

DumpGcovData  latencytop.gcda  latencytop.o

latencytop.c  latencytop.gcno  Makefile

./sane_schedstat:

Makefile          sane_schedstat.gcda  sane_schedstat.o

sane_schedstat.c  sane_schedstat.gcno

./showmap:

Makefile  showmap.c  showmap.o

root@babu-VirtualBox:~/gcov_tests# 

• In case the test machine is different from the build machine, then an additional step of bringing the run time gcov information from test machine to build machine is required. I have automated this step using a script - https://github.com/babuneelam/gcov_scripts/blob/master/fetch_n_merge_gcov. One can use this script from build machine in any build sub-directory to update corresponding directory's run time coverage info from the test machine. Below is the usage info of this script. Use this with "--remote --uinfo" options. I am not doing this step now as my test & build machines are the same.

root@babu-VirtualBox:~/gcov_tests/lkm# ./fetch_n_merge_gcov 

Usage: fetch_n_merge_gcov --[local/remote] --kinfo --uinfo
                    [--kp <kdir_prefix>] [--up <udir_prefix>]

Execute this script from any build sub-directory to update
corresponding run time coverage info

root@babu-VirtualBox:~/gcov_tests/lkm#

• The, from my top build directory, I ran the following lcov commands:

root@babu-VirtualBox:~/gcov_tests# geninfo  . -o ./coverage.info

Found gcov version: 4.8.1

Scanning . for .gcda files ...

Found 2 data files in .

Processing latencytop/latencytop.gcda

Processing sane_schedstat/sane_schedstat.gcda

Finished .info-file creation

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# ls

bin  coverage.info  dump_gcov  latencytop  Makefile  sane_schedstat  showmap

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# genhtml -o ./lcov_data/ ./coverage.info

Reading data file ./coverage.info

Found 2 entries.

Found common filename prefix "/home/babu"

Writing .css and .png files.

Generating output.

Processing file gcov_tests/latencytop/latencytop.c

Processing file gcov_tests/sane_schedstat/sane_schedstat.c

Writing directory view page.

Overall coverage rate:

  lines......: 56.8% (155 of 273 lines)

  functions..: 69.6% (16 of 23 functions)

  branches...: 43.7% (52 of 119 branches)

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# ls -LR lcov_data/

lcov_data/:

amber.png    gcov_tests  index-sort-b.html  ruby.png

emerald.png  glass.png   index-sort-f.html  snow.png

gcov.css     index.html  index-sort-l.html  updown.png

lcov_data/gcov_tests:

latencytop  sane_schedstat

lcov_data/gcov_tests/latencytop:

index.html         index-sort-l.html              latencytop.c.gcov.html

index-sort-b.html  latencytop.c.func.html

index-sort-f.html  latencytop.c.func-sort-c.html

lcov_data/gcov_tests/sane_schedstat:

index.html         sane_schedstat.c.func.html

index-sort-b.html  sane_schedstat.c.func-sort-c.html

index-sort-f.html  sane_schedstat.c.gcov.html

index-sort-l.html

root@babu-VirtualBox:~/gcov_tests# 

• I used firefox to view code coverage information of my tests. 

root@babu-VirtualBox:~/gcov_tests# firefox ./lcov_data/index.html

• For reference, I hosted my code coverage results on my google drive - https://googledrive.com/host/0B4xkNcpT6Zbdb2gzZUlfOURJcTg/. Here is the top level screen:  

• Among the three applications I have, lcov report included only those applications instrumented with gcov. So, engineers should NOT be hasty in concluding the coverage information based on just lcov report, but should cautiously check whether all the code expected to be compiled with gocv flags is compiled so. Any code that is not compiled with gcov flags should be categorized "unknown coverage".

Hope this helps.

gcov/lcov - C & Linux based Dynamic Code Coverage Tools

Gcov & Lcov - Overview

gcov is a GNU code coverage tool. gcov provides three basic functionalities:

• Compile time instrumentation facility: 
• This functionality is built into gcc & can be enabled with a couple of gcc options. So, if you have gcc installed, you get this portion automatically.  
• Run time coverage data collection engine: 
• For user space applications, a gcc packaged library "libgcov" provides this functionality & hence should be linked with applications. As this library comes pre-installed with gcc, you get this portion too automatically. 
• However, for kernel space applications, data collection engine comes through additional kernel code which can be enabled by turning on a compile time kernel flags. So, to get run time coverage info for linux kernel or kernel modules, linux kernel should be recompiled with these additional kernel compilation flags.
• Code coverage reporting Interface:  
• For this functionality, GNU provides a command line utility 'gcov'. However, gcov's cli based user interface is not intuitive. 
• For a more intuitive and GUI based interface, lcov can be used to interpret gcov's coverage information. Let me also delve a bit into lcov's origins. LTP(Linux Test Project), an open source initiative, developed lcov. LTP's goal is to develop a comprehensive test suite for Linux OS. To identify areas of improvement in LTP test suite, LTP uses code coverage as one of the criteria. To help aid in code coverage analysis, LTP developed lcov tool by extending gcov.

gcov is pretty easy to use & gives great results. 
 
User Space Applications - Usage Steps 

Here are the usage steps:

• Instrument Application(s) Code: 
• Recompile the sources with gcov compilation flags: -fprofile-arcs -ftest-coverage. Link the binaries too with LDFLAGS: -fprofile-arcs -ftest-coverage. 
• These two flags are synonymous with "--coverage" flag and also imply "-lgcov" during linking. 
• libgcov provides the gcov's profiling functionality for user space applications.In my VM, I found libgcov installed at /usr/lib/gcc/i686-linux-gnu/4.8/64/libgcov.a, /usr/lib/gcc/i686-linux-gnu/4.8/libgcov.a and /usr/lib/gcc/i686-linux-gnu/4.8/x32/libgcov.a. So, as I mentioned earlier, libgcov comes along with gcc installation.
• Post-compilation & linking, a .gcno file is created in the respective build directory for every c file instrumented with gcov.

• Execute Application(s)' Test Suite(s)
• Fetch run time coverage info: After the test case is executed, dump the gcov run time time coverage information. 
• Run time coverage information is not dumped until the process is terminated. However, most of the embedded applications are background daemons which never terminate. So, I wrote a custom script - https://github.com/babuneelam/gcov_scripts/blob/master/dum_gcov. This script basically attaches to the process through gdb & flushes gcov run time data & then detaches from gdb. 
• For every c file instrumented with gcov, a .gcda file is generated at run time.
• Merge run time coverage info to Application(s)' build directories: 
• For environments where build and testing are done in the same device, this step can be skipped as gcov dumps each application's run time data automatically into corresponding application's build directory. 
• In case of embedded systems, the build server would be different from test machine. So, the gcov run time information should be copied from test machine to build server into applications' build directories. I have automated the steps of extraction & merge using a script - https://github.com/babuneelam/gcov_scripts/blob/master/fetch_n_merge_gcov. One can use this script from build machine in any build sub-directory to update corresponding directory's run time coverage info from the test machine. 
• Convert raw gcov coverage info into lcov's format: At top build directory, run lcov tools - geninfo & genhtml. These tools convert raw gcov coverage information to such a format that a browser can be used present the data to the user.
• Create intuitive Code coverage report: Any web browser can be used to view lcov data as a report. lcov report includes only those applications instrumented with gcov. So, engineers should NOT be hasty in concluding the coverage information based on just lcov report, but should cautiously check whether all the code expected to be compiled with gocv flags is compiled so. Any code that is not compiled with gcov flags should be categorized "unknown coverage".

Kernel or Kernel modules - Usage Steps

Here are the usage steps:

• Enable run time coverage data collection engine:
• To get code coverage for kernel modules, recompile the kernel with CONFIG_GCOV_KERNEL=y. To get code coverage even for entire linux kernel, recompile with CONFIG_GCOV_PROFILE_ALL=y. 
• Instrument Application(s) Code: 
• Recompile the kernel files/modules for which code coverage is sought. To recompile, follow the instructions from https://www.kernel.org/doc/Documentation/gcov.txt copy pasted below:

To enable profiling for specific files or directories, add a line similar to the following to the respective kernel Makefile:         

               For a single file (e.g. main.o):                 

                       GCOV_PROFILE_main.o := y         

               For all files in one directory:

                       GCOV_PROFILE := y 

To exclude files from being profiled even when CONFIG_GCOV_PROFILE_ALL is specified, use:

                       GCOV_PROFILE_main.o := n        

               and:

                       GCOV_PROFILE := n        

• Post-compilation & linking, a .gcno file is created in the respective build directory for every c file instrumented with gcov.
• Execute Application(s)' Test Suite(s)
• Fetch run time coverage info: After the test suite(s) is executed, fetch run time coverage information: 
• Run time coverage information of linux kernel or kernel modules can be accessed any time in debugfs at /sys/kernel/debug/gcov/. 
• For every c file instrumented with gcov, a .gcda file is generated at run time. For a specific file, look for /sys/kernel/debug/gcov/path/to/compile/dir/file.gcda.
• Merge run time coverage info to Application(s)' build directories: 
• Run time coverage should be merged with the kernel or kernel module's build directory. Additionally, in case of embedded systems, data should be extracted from test machine to build machine. 
• I have automated the steps of extraction & merge using a script - https://github.com/babuneelam/gcov_scripts/blob/master/fetch_n_merge_gcov. One can use this script from build machine in any build sub-directory to update corresponding directory's run time coverage info from the test machine. This works for both local test machine scenario or remote test machine scenario.
• Convert raw gcov coverage info into lcov's format: At top build directory, run lcov tools - geninfo & genhtml. These tools convert raw gcov coverage information to such a format that a browser can be used present the data to the user.
• Create intuitive Code coverage report: Any web browser can be used to view lcov data as a report. lcov report includes only those applications instrumented with gcov. So, engineers should NOT be hasty in concluding the coverage information based on just lcov report, but should cautiously check whether all the code expected to be compiled with gocv flags is compiled so. Any code that is not compiled with gcov flags should be categorized "unknown coverage".

Hope this helps.


References:

• Kernel gcov usage: https://www.kernel.org/doc/Documentation/gcov.txt
• gcov man page: http://linux.die.net/man/1/gcov
• lcov man page: http://linux.die.net/man/1/lcov

Dynamic Code Coverage Basics

When I worked with Cisco's product development, I used an in-house code coverage tool. Recently, I also came across c/linux based code coverage tools - gcov/lcov. In this post, I will cover basics of Dynamic Code Coverage.

Purpose of Dynamic Code Coverage 

Dynamic Code coverage tools help to understand the amount of code executed when a test/test suite is executed. Using the code coverage information, the effectiveness of test/ test suite can be improved through test suite optimizations. By achieving the code coverage standards an organization sets internally for itself, the confidence in the product/feature quality is improved. Code coverage is typically used as a pre-condition at three stages of product development cycle:

• Engineering releases typically use certain coverage criteria as a pre-condition before a product/release is rolled out to customers.
• When new features are developed internally, new feature testing & automation engineers use code coverage to prove the effectiveness of the test suite used for testing the feature. Usually, meeting an internal code coverage standard would be a pre-condition to move the feature from experimental branch to a mainline branch & add the respective test suite test suite to regression test suites.
• Development engineers too use code coverage during their functional/unit testing so that they can proactively ensure the quality of their code deliverable to QA teams.

So primarily , code coverage is a Quality Assurance tool.

Secondarily, code coverage can help in understanding the execution frequency of different code blocks & hence provide an opportunity to profile/optimize the code.

Typical Steps in Obtaining Dynamic Code Coverage

• Instrument code: The code for which coverage info is sought is instrumented at compile time: Every line of source code is tracked using a counter to count the number of times the line is executed/traversed, every branch is tracked with branch path tracking counter, etc etc..
• Execute test suite(s): When such instrumented code is tested, runtime coverage information is generated during or after completion of execution.
• Collate all pieces of code coverage info: source code, compile time instrumentation info, runtime coverage info.
• Create intuitive code coverage report: In this step, code coverage reporting tools combine knowledge of code, compile time instrumentation information and runtime coverage information to present code coverage details in an intuitive manner.
• Evaluate against organizational code coverage standards: If coverage standard are met, then move on to next stage in product development cycle. If not, improve test suites, Go back to Step1.

Dynamic Code Coverage Information - Categories

Following is some possible coverage information:

• Statement coverage: Is every statement covered?
• Condition coverage: Is every boolean condition evaluated to true/false?
• Branch coverage: Is every branch/path if/else and switch/case covered?
• Decision coverage: Is every decision covered?
• Function Coverage: Is every function covered?
• Function Entry/Exit coverage: Is every path of entering a function covered? Is every path of exiting a function covered?

Among these, the typically used information: statement coverage, condition coverage, branch coverage and decision coverage.

Misconception about Dynamic Code Coverage

Engineering and QA teams might show a tendency to perceive the covered code as COMPLETELY safe or bug/error free. However, this perception is incorrect because doesn't imply testing logical aspects of the code.

The appropriate way of interpreting dynamic code coverage is

• to focus on uncovered code to verify the adequacy of test suites & improve test suites for increased coverage. 
• to treat covered code information as just one input in assuring code quality and continue to leverage other available methods/techniques for further improvement in code quality

More..
In my upcoming posts, I am covering gcov/lcov - C & Linux based Dynamic Code Coverage Tools, How to - gcov/lcov for linux user space process and How to - gcov/lcov for linux kernel modules.


Hope this helps.

Static Analysis - Overview

Static analysis tools are used to improve software quality proactively by development and QA teams.

Traditional static analysis technology (that early stage tools such as lint used to offer) used to do simple type checking & more over dumping a lot of false positives.

However, advanced/latest static analysis technology finds more valuable defects such as system crashes, memory leaks, uninitialized access, holding reference to freed memory, concurrency issues etc. Even the false positives concern seems to have been addressed in these tools. In addition, static analysis tools seem to provide some software metrics that can help team to proactively publish the quality of their code.

How do advanced static analysis tools detect bugs?

At a very high level, static analysis tools analyze various possible control flows & various possible contexts (variables) using an abstract machine. They also seem to hypothetically replace returning values of various functions/sub-program to evaluate the overall software behavior. They also seem to eliminate exploration of infeasible paths through advanced techniques. Analysis techniques are language-specific, but independent of hardware/OS used to run the software.

Each tool has different techniques & each one will find different defects on the same code base. So, there is not general standard technique used across all the tools.

At a very broad level, advanced static analysis tools can be classified into two categories:

• Sound tools: These tools sacrifice maximization of finding bugs to minimizing false positives to a reasonable level. So, these tools are intentionally imprecise & undecidable - this means possible false positives & false negatives.
• Unsound tools/Formal tools: These tools report all defects checked for & produces no false negatives. These are relatively slower. but, produces a large amount of false positives.

For non-mission-critical commercial applications, deploying sound tools result in diminishing returns for the investment made. So, a combination of "static analysis + other run time testing" is more fetching for non-mission-critical commercial applications.

What kind of defects do static analysis tools detect?

• Resource leaks - memory, socktets, fds
• memory corruption possibilities: buffer overflows, uninitialized access, use after free(holding reference to freed memory), invalid free (null/unallocated)
• Illegal operations: divide by zero, and illegally dereferenced pointers, double free, returning pointer to local variable, negative array index read, 
• concurrency issues (double locking, missing unlocking, lock ordering issues (deadlocks), blocking calls/sleep in lock acquired state)
• optimizations such as dead code
• portability to other architectures
• missing returns, unprocessed return values, missing break in switch
• large pass by value function parameters, large stack use
• security vulnerabilities (buffer overruns, system calle based or exec based command injection, 

Popular products in Static Analysis

• Coverity Prevent from Coverity (unsound tool) - C/C++/Java, commercial
• KlockWork K7/Insight (unsound tool) - C/C++/Java, commercial, 
• CodeSonar from GammTech (unsound tool - C/C++, commercial
• Mathworks PolySpace (sound/formal tool) - commercial
• cppcheck - C++, free tool, light weight
• Lint - C/C++, open source, light weight
• PC-Lint - C, commercial, heard of generating a lot of false positives.
• FlexLink - closed source, lightweight tool compared to coverity/KlockWork
• Splint - C, free, based on lint with improvements in security/style checking, light weight

What stages of product development cycle should these tools be deployed?

• Run by developers on the diff before committing to version control system.
• Run by development & QA teams in nightly builds before & after internal release to QA.

Does defect detection using static analysis make QA testing and other run time tools/techniques (such as dynamic analysis) unnecessary?

No. Static analysis doesn't guarantee absence of defects. They sacrifice maximizing the defect detection to generating increased false positives. Also, the tests static analysis tools run are not real functional/practical tests customers expect the software to use for.

For this reason, the real QA testing as well as the need for other run time tools to debug issues found in QA testing is very much necessary. So, static analysis tools is only additional leverage rather than either/or comparison wit QA testing and run time tools.

Static analysis tools vs dynamic analysis tools

  

Static analysis tools compared to dynamic analysis tools:

• Static analysis doesn't not need test cases or real test hardware to cull out the defects. 
• Static analysis can analyze even parts of code practical test cases can't reach or code reviewers can spot.. so, makes static analysis a must-have tool for mission-critical software such as ...
• static analysis visualizes every possible control flows & detect defects where as dynamic analysis tools report only defect pertaining to the execution path of the test being performed.
• Dynamic analysis helps in functional tests as opposed to static analysis' hypothetical tests. 
• Static analysis is faster.. does more tests than dynamic analysis in the same period of time particularly due to lack of need to run real tests.

Pros:

• Static analysis tools perform source code analysis without running it. So,
• no test team or test hardware is necessary.
• a lot of corner case tests which might be impossible or difficult to test would be exercised.
• Cheaper compared to depending on QA teams for all bugs. So, proactively using static analysis tool for basic bugs and depending on QA teams for advanced bugs is more cost effective.
• Static analysis can be done at any stage of product development cycle even before a usable product is available. So, the bugs caught at early stages of development offers planning buffer & avoid lots of debugging troubles later on.

Cons:

• Tests not related to functional requirements, integration testing requirements
• Shift of developer cycles from programming to dealing with false positives

TBD:

• Can we use static analysis defect to improve QA test suites?
• What is the approx time these tools takes for certain number of lines..1,400 lines of code per minute?