= Improve Coverage Analysis Tools = [[TOC(GSoC/2017/coveragetools , depth=2)]] '''Student:''' Cillian O'Donnell. '''Mentors:''' Joel Sherrill, Kuan Hsun Chen. '''Introduction:''' This project will switch the RTEMS Source Builder from QEMU to Couverture-QEMU. It will integrate Couverture-QEMU and the scripts driving it into RTEMS Tester framework, converting the BASH scripts to Python. The coverage report tool Covoar written in C++ will then be modified to generate XML output. '''Previous work:''' There had been 2 previous student work for SOCIS in 2014[1] and 2015[2]. The series of patches detailed in their blogs are the starting point of this project. As years of development has taken place since their work, not all the patches still applied. When I pieced them back together, there were build errors in Covoar (the RTEMS coverage reporting tool) and its associated files. In 2015 this setup produced coverage reports for leon3 and pc386, so the first thing to be done is to revive this setup and generate the Leon3 report again. '''Related Open Project Tickets:''' [https://devel.rtems.org/ticket/3037 Ticket-#3037: QEMU fails to exit after test, 4 BSPS] [https://devel.rtems.org/ticket/3039 Ticket-#3039: ppc/qemuprep: Qemu fails to boot, no boot partition found] = Development Blog = [http://summerofrtems.blogspot.ie/2017/05/introduction.html Introduction] [https://cillianodonnell.github.io//rtems/2017/06/03/reviving-work.html Reviving-RTEMS-Tester-Work] [https://cillianodonnell.github.io//rtems,/testing/2017/06/12/testing-qemu.html Testing-Couverture-Qemu] [https://cillianodonnell.github.io//rtems,/rsb/2017/06/22/RSB-qemu.html Switching-RSB-to-Couverture-Qemu] [https://cillianodonnell.github.io//rtems,/gsoc/2017/07/26/phase-2-update.html Phase-2-Update] [https://cillianodonnell.github.io//rtems,/gsoc/2017/08/16/jump-tables.html Jump-Tables-a-Recurring-Problem] = Building Couverture-Qemu with the RSB = The build set for Couverture-Qemu can be found in rsb/bare/config/devel and the command to build it from the top level of the RSB source is: {{{ $ cd rtems $ ../source-builder/sb-set-builder \ --log=couverture_qemu_log.txt \ --prefix=$HOME/development/4.12 \ devel/couverture-qemu }}} This will build the qemu tools for the currently supported architectures a list of which can be found in rsb/source-builder/config/couverture-qemu-2-1.cfg. The current list at the time of writing is: {{{ 22 %define qemu_archs arm-softmmu,i386-softmmu,lm32-softmmu,mips-softmmu 23 %define qemu_archs %{qemu_archs},ppc-softmmu,sparc-softmmu }}} There is also a feature to override this list and build the architectures of your choice, if needed for a new bsp maybe. The option is --with-qemu-archs= which is passed in the command line argument to invoke the RSB {{{ $ ../source-builder/sb-set-builder \ --log=couverture_qemu_log.txt \ --prefix=$HOME/development/4.12 \ --with-qemu-archs=ppc-softmmu \ devel/couverture-qemu }}} This would build ppc-softmmu only and place it in your prefix location. = Coverage Analysis in RTEMS Tester = == Symbol Set Configurtion == The tests run in rtems-test will have there trace data logged in a file called .exe.cov. This will be used to check what percentage of the symbols they cover and the symbols to be analysed are in the symbol set config file which can be found here {{{ From top directory of rtems-tools $ cd tester/rtems/testing/coverage $ ls Categories.txt SPARC-Annul-Slot-Explanation.txt symbol_sets.cfg Explanations.txt style.css }}} The format of the config file is {{{ symbolset: name = NAME_OF_SET lib = ABSOLUTE_PATH_OF_LIBRARY_1 lib = ABSOLUTE_PATH_OF_LIBRARY_2 .. .. .. lib = ABSOLUTE_PATH_OF_LIBRARY_N symbolset: name = NAME_OF_ANOTHER_SET lib = ... }}} 1. Each set is defined with the keyword 'symbolset:' 2. name = ... defines a suitable descriptive name for the collections of libraries. 3. lib = ... to add a library of symbols to be used. An example of this is format is the score set for Leon 3. {{{ symbolset: name=score lib=sparc-rtems4.12/c/leon3/cpukit/score/libscore.a }}} == Supported BSPS == Leon 3: Use '--rtems-bsp=leon3-qemu' with rtems-test cmd. == Performing Coverage Analysis with rtems-test command == The coverage analysis is performed by adding the --coverage flag when running the rtems-test cmd and selecting a bsp config file that supports coverage. Also the bsp build directory is now added as a separate option that can be used by the symbol set file so the length of the file names can be kept down in symbol_set.cfg file. {{{ // Create a suitable directory to hold coverage reports and associated files. $ cd $ mkdir coverage_test $ cd coverage_test // Now run the rtems-test cmd with --coverage $ $HOME/development/rtems/test/rtems-tools/tester/rtems-test / --rtems-tools=$HOME/development/rtems/4.12 / --log=coverage-analysis.log / --rtems-bsp=leon3-qemu / --coverage / --rtems-builddir=$HOME/development/rtems/leon3 / sparc-rtems4.12/c/leon3/testsuites }}} After the testsuite is run, covoar then performs multiple runs analysing the trace data collected by Couverture-QEMU for all tests on each symbol set in symbol_set.cfg. Covoar sections the objdump into seperate functions and uses the trace data from Couverture-Qemu that records trace blocks as 'executed' and 'not executed' to mark branches as 'taken' and 'not taken'. Covoar then creates coverage maps from the results. The maps are then merged together and a report is generated. The report is a html file that can be found in the directory created above coverage_test called report.html. {{{ // End of the output of the RTEMS testsuite run with rtems-test ... .... [565/565] p:545 f:1 u:4 e:0 I:0 B:3 t:5 i:3 | sparc/leon3: tmtimer01.exe Passed: 549 Failed: 1 User Input: 4 Expected Fail: 0 Indeterminate: 0 Benchmark: 3 Timeout: 5 Invalid: 3 Total: 565 Average test time: 0:00:01.612648 Testing time : 0:15:11.146458 Linking executables to /home/cpod/coverage_test/leon3/coverage Symlinks made Running covoar for score covoar -C/home/cpod/coverage_test/leon3/coverage/config -S /home/cpod/coverage_test/leon3/coverage/score.symcfg -O /home/cpod/coverage_test/leon3/test/score /home/cpod/coverage_test/leon3/coverage/*.exe Reading symbols sets configuration for symbol set file: /home/cpod/coverage_test/leon3/coverage/score.symcfg Generating symbol file for score Coverage run for score finished successfully. ----------------------------------------------- Generating reports Cleaning workspace up Coverage analysis finished. You can find results in /home/cpod/coverage_test/leon3 }}} = References = [1] http://kmiesowicz.blogspot.ie/2014/08/new-approach-to-coverage-analysis-in.html [2] http://socis2015rtems.blogspot.ie/