gcov run-time data format change - impact on gcov usage

In this post, I will cover gcov run time data formats & the impact of this on gcov usage.

gcov run-time data format change

The first time I tried getting code coverage for a kernel module, I didn't get any gcda files in /sys/kernel/debug/gcov. Only a reset file was present. When I debugged the issue, I found the root cause - an issue arising from gcov's run time data format change introduced in gcc 4.7. 

A new gcov run time data format is introduced in gcc 4.7. The new format has more efficient data structures used for profiling than the earlier format. 

Impact on gcov usage

Due to this format change, run time coverage information collection can be broken.

The application code instrumentation facility is provided by gcc's compile flags. The gcov run time data collection engine is provided by libcov in case of user space applications or additional kernel code in case of kernel modules.

To accommodate gcov run time format change, both gcc version as well as libgcov/kernel-gcov-code should be moved together. Otherwise, run time coverage information collection can be broken due to incompatibility between instrumented code & the gcov run time data collection engine.

• Impact on User space Applications: As libgcov is part of gcc installation, we can safely assume that both gcc's instrumentation capability as well as gcov run time data collection engine are always moved together. So, one will not see any impact while using gcov on user space applications.
• Impact on Kernel modules or Kernel: In case of kernel, kernel-gcov-code provides gcov run time data collection engine. The support for newer gcov run time format is added to kernel in 3.13 version. And the gcc version installed on the system is controlled by user. So, there is a significant possibility that gcc version & kernel-gcov-code become incompatible. Here are the possibilities:

   
Scenario1 : gcc version < 4.7, kernel-gcov-code < 3.11: compatible formats

Scenario 2: gcc version < 4.7, kernel-gcov-code > 3.11: incompatible, but linux kernel has backward compatibility for past gcc versions. Selet "Auto detect"..

Scenario 3 : gcc version >=4.7, kernel-gcov-code < 3.11: incompatible formats, run time data collection broken

Scenario 4: gcc version>=4.7, kernel version>3.11: compatible formats

I hit scenario 3. I solved it by upgrading my kernel to 3.13.5. Before recompiling 3.13.5 kernel, I selected "Autodetect Gcov format" in "make menuconfig". The exact steps are detailed in my other post at How to - Linux Kernel Compilation in Ubuntu VM/VirtualBox.

May be, gcc could have provided a facility to compile with earlier gcov formats. 

How to find your gcc version?

Refer my other post at  My cheatsheet for gcc

How to find your linux kernel version?

Refer "OS Info" section in my other post at Linux System Info Commands

Hope this helps.

Merging multiple coverage files using lcov

In my earlier posts How to - gcov/lcov for linux user space process and How to - gcov/lcov for linux kernel modules, I covered how to get a code coverage report for user space applications and kernel modules. In this post, I will provide a quick tip for merging multiple coverage files.

When do we need to merge coverage (*.gcda) files?

• In High availability/Clustering environments, we get coverage information from multiple nodes. Sometimes, developers would like to combine coverage info from all the nodes & view the merged report rather than analyze multiple reports.

Steps to collect & merge coverage info

Lets take this sample program:

#include <stdio.h>

int main ( int argc, char *argv[] )
{
    if ( argc != 2 ) /* argc should be 2 for correct execution */
    {
        /* We print argv[0] assuming it is the program name */
        printf( "usage: %s test_num", argv[0] );
    }
    else
    {
        int i = atoi(argv[1]);

        if (i == 1)
                printf("Testcase 1\r\n");
        else
                printf("Testcase 2\r\n");
    }
}

I ran two tests on this sample program, collected gcda files & merged them using lcov:

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c

root@babu-VirtualBox:~/gcov_tests/merge_tests# gcc -fprofile-arcs -ftest-coverage a.c

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c  a.gcno  a.out

root@babu-VirtualBox:~/gcov_tests/merge_tests# ./a.out 1

Testcase 1

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c  a.gcda  a.gcno  a.out 

root@babu-VirtualBox:~/gcov_tests/merge_tests# geninfo  . -o ./coverage1.info

Found gcov version: 4.8.1

Scanning . for .gcda files ...

Found 1 data files in .

Processing a.gcda

Finished .info-file creation

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c  a.gcda  a.gcno  a.out  coverage1.info 

root@babu-VirtualBox:~/gcov_tests/merge_tests# rm a.gcda

root@babu-VirtualBox:~/gcov_tests/merge_tests# ./a.out 2

Testcase 2

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c  a.gcda  a.gcno  a.out  coverage1.info

root@babu-VirtualBox:~/gcov_tests/merge_tests# geninfo  . -o ./coverage2.info

Found gcov version: 4.8.1

Scanning . for .gcda files ...

Found 1 data files in .

Processing a.gcda

Finished .info-file creation

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c  a.gcda  a.gcno  a.out  coverage1.info  coverage2.info

root@babu-VirtualBox:~/gcov_tests/merge_tests# mkdir lcov_data

root@babu-VirtualBox:~/gcov_tests/merge_tests# lcov --add-tracefile coverage2.info -t test2 -a coverage1.info -t test1 -o coverage_merged.info

Combining tracefiles.

Reading tracefile coverage2.info

Reading tracefile coverage1.info

Writing data to coverage_merged.info

Overall coverage rate:

  lines......: 87.5% (7 of 8 lines)

  functions..: 100.0% (1 of 1 function)

  branches...: 75.0% (3 of 4 branches)

root@babu-VirtualBox:~/gcov_tests/merge_tests# ls

a.c     a.gcno  coverage1.info  coverage_merged.info  lcov_data

a.gcda  a.out   coverage2.info  

root@babu-VirtualBox:~/gcov_tests/merge_tests# genhtml -o ./lcov_data/ ./coverage_merged.info

Reading data file ./coverage_merged.info

Found 1 entries.

Found common filename prefix "/home/babu"

Writing .css and .png files.

Generating output.

Processing file gcov_tests/merge_tests/a.c

Writing directory view page.

Overall coverage rate:

  lines......: 87.5% (7 of 8 lines)

  functions..: 100.0% (1 of 1 function)

  branches...: 75.0% (3 of 4 branches)

root@babu-VirtualBox:~/gcov_tests/merge_tests# 

I hosted my code coverage results on my google drive - https://googledrive.com/host/0B4xkNcpT6ZbddzlPMHp3bEV3dk0/. Here is the screenshot that shows coverage for a.c:

  

As we can see both the lines which prints" Testcase 1" & "Testcase 2" are shown as covered lines in the merged report. 

What are the caveats when interpreting a merged coverage report?

• TBD

References:

http://linux.die.net/man/1/lcov

How to - gcov/lcov for linux kernel modules

To test & explore gcov/lcov functionality for kernel modules, here are the steps I used:

• I wanted to get coverage for two simple kernel modules - character driver with a user space counterpart & a hello LKM. Here is the code: https://github.com/babuneelam/gcov_kspace_tests.

• Re-compile the kernel with GCOV functionality enabled. To enable Gcov functionality in kernel, enable CONFIG_DEBUG_FS, CONFIG_GCOV_KERNEL, CONFIG_GCOV_FORMAT_AUTODETECT and CONFIG_GCOV_PROFILE_ALL (if entire linux kernel requires coverage). If "make menuconfig" is ued, the following screens screens help to enable gcov in kernel:

       I followed http://techvolve.blogspot.com/2014/02/how-to-linux-kernel-compilation.html to recompile. 

• To instrument the code with gcov, I enabled gcov compilation flags in the makefile of simple_krn_drv & simple_test_app, but left out hello_lkm. I also turned off any gcc optimization by removing "O" flags :

root@babu-VirtualBox:~/gcov_tests/lkm# grep -irne "-fprofile-arcs" *
simple_krn_drv/Makefile:5:CFLAGS=-ftest-coverage -fprofile-arcs  --disable-initfini-array 
simple_test_app/Makefile:6:CFLAGS       = -fPIC -pedantic -fprofile-arcs -ftest-coverage -march=native -ggdb3

root@babu-VirtualBox:~/gcov_tests/lkm# 

• I compiled all the kernel modules & the user space test application. 

• After compilation, I see that gcc generated .gcov files in the two kernel modules instrumented with gcov flags:

root@babu-VirtualBox:~/gcov_tests/lkm# find . -type f -name "*.gcno"
./simple_krn_drv/.tmp_simple_dev.gcno
./simple_krn_drv/simple_dev.mod.gcno
./simple_test_app/simple_hello.gcno
root@babu-VirtualBox:~/gcov_tests/lkm# 

• The gcov run time date is not available before I inserted my kernel modules: 

                  root@babu-VirtualBox:~/gcov_tests/lkm# ls /sys/kernel/debug/gcov/
                  reset  usr  var
                  root@babu-VirtualBox:~/gcov_tests/lkm# 

• Test modules: Insert LKMs, run the simple_test_app

• Merge run time coverage information from debugfs to build directory. 
• I have automated this step using a script tool - https://github.com/babuneelam/gcov_scripts/blob/master/fetch_n_merge_gcov. One can use this script tool on build machine in any build sub-directory to update corresponding directory's run time coverage info from the test machine. Below is the usage info of this script. 

root@babu-VirtualBox:~/gcov_tests/lkm# ./fetch_n_merge_gcov 

Usage: fetch_n_merge_gcov --[local/remote] --kinfo --uinfo
                    [--kp <kdir_prefix>] [--up <udir_prefix>]

Execute this script from any build sub-directory to update
corresponding run time coverage info

root@babu-VirtualBox:~/gcov_tests/lkm#

• As my test & build machine are both same, I used this with "--local --kinfo" options.

root@babu-VirtualBox:~/gcov_tests/lkm# ./fetch_n_merge_gcov --local --kinfo

Is /home/babu/gcov_tests/lkm the build sub-directory you want to update run time coverage info with? : (y/n)?y

Deleting old run time runtime info in build directory: /home/babu/gcov_tests/lkm

/home/babu/gcov_tests/lkm/simple_krn_drv/simple_dev.gcda

/home/babu/gcov_tests/lkm/simple_test_app/simple_hello.gcda

Importing new kspace coverage run time info from test device....

Treating /sys/kernel/debug/gcov/ as top level kspace gcov directory in test machine

mount: none already mounted or /sys/kernel/debug/ busy

mount: according to mtab, none is already mounted on /sys/kernel/debug

Merging new kspace coverage run time info to the build directory....

/home/babu/gcov_tests/lkm/simple_krn_drv/simple_dev.gcda

root@babu-VirtualBox:~/gcov_tests/lkm#

• In case the test machine is different from the build machine, use this script with "--remote --kinfo" options.

• Then, from my top build directory, I ran the following lcov commands:

root@babu-VirtualBox:~/gcov_tests/lkm# geninfo  . -o ./coverage.info

Found gcov version: 4.8.1

Scanning . for .gcda files ...

Found 2 data files in .

Processing simple_krn_drv/simple_dev.gcda

Cannot open source file include/linux/fs.h

Processing hello_lkm/hello.gcda

Finished .info-file creation

root@babu-VirtualBox:~/gcov_tests/lkm#

root@babu-VirtualBox:~/gcov_tests/lkm# genhtml -o ./lcov_data/ ./coverage.info

Reading data file ./coverage.info

Found 4 entries.

Found common filename prefix "/home/babu/gcov_tests/lkm/simple_krn_drv"

Writing .css and .png files.

Generating output.

Processing file /home/babu/gcov_tests/lkm/hello_lkm/hello.c

Processing file simple_dev.c

Processing file include/linux/fs.h

Processing file /usr/src/linux-3.13.5/arch/x86/include/asm/uaccess.h

Writing directory view page.

Overall coverage rate:

 lines......: 23.1% (9 of 39 lines)

 functions..: 28.6% (2 of 7 functions)

 branches...: 11.1% (1 of 9 branches)

root@babu-VirtualBox:~/gcov_tests/lkm#

• I used firefox to view code coverage information of my tests. 

root@babu-VirtualBox:~/gcov_tests# firefox ./lcov_data/index.html

• For reference, I hosted my code coverage results on my google drive - https://googledrive.com/host/0B4xkNcpT6ZbdaEJrelEtTk1uODA/. Here is the top level screen:  

                                          

• Among the two kernel modules I have, lcov report included only one module instrumented with gcov. So, engineers should NOT be hasty in concluding the coverage information based on just lcov report, but should cautiously check whether all the code expected to be compiled with gocv flags is compiled so. Any code that is not compiled with gcov flags should be categorized "unknown coverage".

Hope this helps.

How to - gcov/lcov for linux user space process

To test & explore gcov/lcov functionality for user space applications, here are the steps I used:

• I wanted to get coverage for a few android tools that I ported to linux a while ago. Here is the code: https://github.com/babuneelam/gcov_uspace_tests. Here is the structure of my applications' source code:

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  latencytop  Makefile  sane_schedstat  showmap

./bin:

./latencytop:

DumpGcovData  latencytop.c  Makefile

./sane_schedstat:

Makefile  sane_schedstat.c

./showmap:

Makefile  showmap.c

root@babu-VirtualBox:~/gcov_tests# 

• To instrument the code with gcov, I enabled gcov compilation flags in two applications latencytop & sane_schedstat. I also turned off any gcc optimization by removing "O" flags :

root@babu-VirtualBox:~/gcov_tests# grep -irne "-fprofile-arcs" *

latencytop/Makefile:6:CFLAGS       = -fPIC -pedantic -fprofile-arcs -ftest-coverage -march=native -ggdb3

sane_schedstat/Makefile:6:CFLAGS       = -fPIC -pedantic -fprofile-arcs -ftest-coverage -march=native -ggdb3

root@babu-VirtualBox:~/gcov_tests# 

• I compiled all the applications from top directory:

root@babu-VirtualBox:~/gcov_tests# make

• After compilation, I see that gcc generated .gcov files in the two applications instrumented with gcov flags:

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  latencytop  Makefile  sane_schedstat  showmap

./bin:

latencytop  sane_schedstat  showmap

./latencytop:

latencytop.c  latencytop.gcno  latencytop.o  Makefile

./sane_schedstat:

Makefile  sane_schedstat.c  sane_schedstat.gcno  sane_schedstat.o

./showmap:

Makefile  showmap.c  showmap.o

root@babu-VirtualBox:~/gcov_tests# 

• I then executed latencytop & sane_schedstat binaries in two different terminals. Both these applications run in a loop. So, these commands kept looping in those terminals.

          
               

• I then dumped gcov run time date using a custom script I wrote - https://github.com/babuneelam/gcov_scripts/blob/master/dump_gcov. This script basically attaches to the process through gdb & flushes gcov run time data & then detaches from gdb.

root@babu-VirtualBox:~/gcov_tests# ./dump_gcov latencytop  sane_schedstat

Application: latencytop 
Dumping gcov data...
Gcov data dump complete.

Application: sane_schedstat 
Dumping gcov data...
Gcov data dump complete.
  
root@babu-VirtualBox:~/gcov_tests# 

• After this, I see that dump_gcov script generated .gcda files in the two applications instrumented with gcov flags.

root@babu-VirtualBox:~/gcov_tests# ls -LR

.:

bin  dump_gcov  latencytop  Makefile  sane_schedstat  showmap

./bin:

latencytop  sane_schedstat  showmap

./latencytop:

DumpGcovData  latencytop.gcda  latencytop.o

latencytop.c  latencytop.gcno  Makefile

./sane_schedstat:

Makefile          sane_schedstat.gcda  sane_schedstat.o

sane_schedstat.c  sane_schedstat.gcno

./showmap:

Makefile  showmap.c  showmap.o

root@babu-VirtualBox:~/gcov_tests# 

• In case the test machine is different from the build machine, then an additional step of bringing the run time gcov information from test machine to build machine is required. I have automated this step using a script - https://github.com/babuneelam/gcov_scripts/blob/master/fetch_n_merge_gcov. One can use this script from build machine in any build sub-directory to update corresponding directory's run time coverage info from the test machine. Below is the usage info of this script. Use this with "--remote --uinfo" options. I am not doing this step now as my test & build machines are the same.

root@babu-VirtualBox:~/gcov_tests/lkm# ./fetch_n_merge_gcov 

Usage: fetch_n_merge_gcov --[local/remote] --kinfo --uinfo
                    [--kp <kdir_prefix>] [--up <udir_prefix>]

Execute this script from any build sub-directory to update
corresponding run time coverage info

root@babu-VirtualBox:~/gcov_tests/lkm#

• The, from my top build directory, I ran the following lcov commands:

root@babu-VirtualBox:~/gcov_tests# geninfo  . -o ./coverage.info

Found gcov version: 4.8.1

Scanning . for .gcda files ...

Found 2 data files in .

Processing latencytop/latencytop.gcda

Processing sane_schedstat/sane_schedstat.gcda

Finished .info-file creation

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# ls

bin  coverage.info  dump_gcov  latencytop  Makefile  sane_schedstat  showmap

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# genhtml -o ./lcov_data/ ./coverage.info

Reading data file ./coverage.info

Found 2 entries.

Found common filename prefix "/home/babu"

Writing .css and .png files.

Generating output.

Processing file gcov_tests/latencytop/latencytop.c

Processing file gcov_tests/sane_schedstat/sane_schedstat.c

Writing directory view page.

Overall coverage rate:

  lines......: 56.8% (155 of 273 lines)

  functions..: 69.6% (16 of 23 functions)

  branches...: 43.7% (52 of 119 branches)

root@babu-VirtualBox:~/gcov_tests# 

root@babu-VirtualBox:~/gcov_tests# ls -LR lcov_data/

lcov_data/:

amber.png    gcov_tests  index-sort-b.html  ruby.png

emerald.png  glass.png   index-sort-f.html  snow.png

gcov.css     index.html  index-sort-l.html  updown.png

lcov_data/gcov_tests:

latencytop  sane_schedstat

lcov_data/gcov_tests/latencytop:

index.html         index-sort-l.html              latencytop.c.gcov.html

index-sort-b.html  latencytop.c.func.html

index-sort-f.html  latencytop.c.func-sort-c.html

lcov_data/gcov_tests/sane_schedstat:

index.html         sane_schedstat.c.func.html

index-sort-b.html  sane_schedstat.c.func-sort-c.html

index-sort-f.html  sane_schedstat.c.gcov.html

index-sort-l.html

root@babu-VirtualBox:~/gcov_tests# 

• I used firefox to view code coverage information of my tests. 

root@babu-VirtualBox:~/gcov_tests# firefox ./lcov_data/index.html

• For reference, I hosted my code coverage results on my google drive - https://googledrive.com/host/0B4xkNcpT6Zbdb2gzZUlfOURJcTg/. Here is the top level screen:  

• Among the three applications I have, lcov report included only those applications instrumented with gcov. So, engineers should NOT be hasty in concluding the coverage information based on just lcov report, but should cautiously check whether all the code expected to be compiled with gocv flags is compiled so. Any code that is not compiled with gcov flags should be categorized "unknown coverage".

Hope this helps.