wiki:GSoC/2017/coveragetools

Improve Coverage Analysis Tools

Student: Cillian O'Donnell.

Mentors: Joel Sherrill, Kuan Hsun Chen.

Introduction: This project will switch the RTEMS Source Builder from QEMU to Couverture-QEMU. It will integrate Couverture-QEMU and the scripts driving it into RTEMS Tester framework, converting the BASH scripts to Python. The coverage report tool Covoar written in C++ will then be modified to generate XML output.

Previous work: There had been 2 previous student work for SOCIS in 2014[1] and 2015[2]. The series of patches detailed in their blogs are the starting point of this project. As years of development has taken place since their work, not all the patches still applied. When I pieced them back together, there were build errors in Covoar (the RTEMS coverage reporting tool) and its associated files. In 2015 this setup produced coverage reports for leon3 and pc386, so the first thing to be done is to revive this setup and generate the Leon3 report again.

Related Open Project Tickets: Ticket-#3037: QEMU fails to exit after test, 4 BSPS Ticket-#3039: ppc/qemuprep: Qemu fails to boot, no boot partition found

Development Blog

Introduction

Reviving-RTEMS-Tester-Work

Testing-Couverture-Qemu

Switching-RSB-to-Couverture-Qemu

Phase-2-Update

Jump-Tables-a-Recurring-Problem

Building Couverture-Qemu with the RSB

The build set for Couverture-Qemu can be found in rsb/bare/config/devel and the command to build it from the top level of the RSB source is:

$ cd rtems
$ ../source-builder/sb-set-builder \
--log=couverture_qemu_log.txt \
--prefix=$HOME/development/4.12 \
devel/couverture-qemu

This will build the qemu tools for the currently supported architectures a list of which can be found in rsb/source-builder/config/couverture-qemu-2-1.cfg. The current list at the time of writing is:

22 %define qemu_archs arm-softmmu,i386-softmmu,lm32-softmmu,mips-softmmu           
23 %define qemu_archs %{qemu_archs},ppc-softmmu,sparc-softmmu 

There is also a feature to override this list and build the architectures of your choice, if needed for a new bsp maybe. The option is --with-qemu-archs=<arch-you-want-to-build> which is passed in the command line argument to invoke the RSB

$ ../source-builder/sb-set-builder \
--log=couverture_qemu_log.txt \
--prefix=$HOME/development/4.12 \
--with-qemu-archs=ppc-softmmu \
devel/couverture-qemu

This would build ppc-softmmu only and place it in your prefix location.

Coverage Analysis in RTEMS Tester

Symbol Set Configurtion

The tests run in rtems-test will have there trace data logged in a file called <test_name>.exe.cov. This will be used to check what percentage of the symbols they cover and the symbols to be analysed are in the symbol set config file which can be found here

From top directory of rtems-tools
$ cd tester/rtems/testing/coverage
$ ls
Categories.txt    SPARC-Annul-Slot-Explanation.txt  symbol_sets.cfg
Explanations.txt  style.css

The format of the config file is

 symbolset:
 name = NAME_OF_SET
 lib = ABSOLUTE_PATH_OF_LIBRARY_1
 lib = ABSOLUTE_PATH_OF_LIBRARY_2
 ..
 ..
 ..
 lib = ABSOLUTE_PATH_OF_LIBRARY_N
symbolset:
 name = NAME_OF_ANOTHER_SET
 lib = ... 
  1. Each set is defined with the keyword 'symbolset:'
  2. name = ... defines a suitable descriptive name for the collections of libraries.
  3. lib = ... to add a library of symbols to be used.

An example of this is format is the score set for Leon 3.

symbolset:                                                                      
    name=score                                                                  
    lib=sparc-rtems4.12/c/leon3/cpukit/score/libscore.a

Supported BSPS

Leon 3: Use '--rtems-bsp=leon3-qemu' with rtems-test cmd.

Performing Coverage Analysis with rtems-test command

The coverage analysis is performed by adding the --coverage flag when running the rtems-test cmd and selecting a bsp config file that supports coverage. Also the bsp build directory is now added as a separate option that can be used by the symbol set file so the length of the file names can be kept down in symbol_set.cfg file.

// Create a suitable directory to hold coverage reports and associated files.

$ cd
$ mkdir coverage_test
$ cd coverage_test


// Now run the rtems-test cmd with --coverage

$ $HOME/development/rtems/test/rtems-tools/tester/rtems-test /
--rtems-tools=$HOME/development/rtems/4.12 /
--log=coverage-analysis.log /
--rtems-bsp=leon3-qemu /
--coverage /
--rtems-builddir=$HOME/development/rtems/leon3 /
sparc-rtems4.12/c/leon3/testsuites

After the testsuite is run, covoar then performs multiple runs analysing the trace data collected by Couverture-QEMU for all tests on each symbol set in symbol_set.cfg. Covoar sections the objdump into seperate functions and uses the trace data from Couverture-Qemu that records trace blocks as 'executed' and 'not executed' to mark branches as 'taken' and 'not taken'. Covoar then creates coverage maps from the results. The maps are then merged together and a report is generated. The report is a html file that can be found in the directory created above coverage_test called report.html.

// End of the output of the RTEMS testsuite run with rtems-test
...
....
[565/565] p:545 f:1   u:4   e:0   I:0   B:3   t:5   i:3   | sparc/leon3: tmtimer01.exe
Passed:        549
Failed:          1
User Input:      4
Expected Fail:   0
Indeterminate:   0
Benchmark:       3
Timeout:         5
Invalid:         3
Total:         565
Average test time: 0:00:01.612648
Testing time     : 0:15:11.146458
Linking executables to /home/cpod/coverage_test/leon3/coverage
Symlinks made
Running covoar for score
covoar -C/home/cpod/coverage_test/leon3/coverage/config -S /home/cpod/coverage_test/leon3/coverage/score.symcfg -O /home/cpod/coverage_test/leon3/test/score /home/cpod/coverage_test/leon3/coverage/*.exe
Reading symbols sets configuration for symbol set file: /home/cpod/coverage_test/leon3/coverage/score.symcfg
Generating symbol file for score
Coverage run for score finished successfully.
-----------------------------------------------
Generating reports
Cleaning workspace up
Coverage analysis finished. You can find results in /home/cpod/coverage_test/leon3

References

[1] http://kmiesowicz.blogspot.ie/2014/08/new-approach-to-coverage-analysis-in.html

[2] http://socis2015rtems.blogspot.ie/

Last modified on 08/19/17 at 15:06:58 Last modified on 08/19/17 15:06:58