Changes between Version 13 and Version 14 of GSoC/2017/coveragetools


Ignore:
Timestamp:
Jul 27, 2017, 8:15:52 PM (2 years ago)
Author:
Cillian O'Donnell
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GSoC/2017/coveragetools

    v13 v14  
    99'''Introduction:''' This project will switch the RTEMS Source Builder from QEMU to Couverture-QEMU. It will integrate Couverture-QEMU and the scripts driving it into RTEMS Tester framework, converting the BASH scripts to Python. The coverage report tool Covoar written in C++ will then be modified to generate XML output.
    1010
     11'''Previous work:''' There had been 2 previous student work for SOCIS in 2014[1] and 2015[2]. The series of patches detailed in their blogs are the starting point of this project. As years of development has taken place since their work, not all the patches still applied. When I pieced them back together, there were build errors in Covoar (the RTEMS coverage reporting tool) and its associated files. In 2015 this setup produced coverage reports for leon3 and pc386, so the first thing to be done  is to revive this setup and generate the Leon3 report again.
     12
    1113'''Related Open Project Tickets:'''
    1214[https://devel.rtems.org/ticket/3037 Ticket-#3037: QEMU fails to exit after test, 4 BSPS]
    13 [https://devel.rtems.org/ticket/3038 Ticket-#3038: QEMU cannot initialize audio driver bsp: lm32/milkymist]
    1415[https://devel.rtems.org/ticket/3039 Ticket-#3039: ppc/qemuprep: Qemu fails to boot, no boot partition found]
    1516
     
    2324[https://cillianodonnell.github.io//rtems,/rsb/2017/06/22/RSB-qemu.html Switching-RSB-to-Couverture-Qemu]
    2425
     26[https://cillianodonnell.github.io//rtems,/gsoc/2017/07/26/phase-2-update.html Phase-2-Update]
     27
    2528= Building Couverture-Qemu with the RSB =
    2629The build set for Couverture-Qemu can be found in rsb/bare/config/devel and the command to build it from the top level of the RSB source is:
    2730{{{
    28 cd rtems
    29 ../source-builder/sb-set-builder \
     31$ cd rtems
     32$ ../source-builder/sb-set-builder \
    3033--log=couverture_qemu_log.txt \
    3134--prefix=$HOME/development/4.12 \
     
    4346
    4447{{{
    45 ../source-builder/sb-set-builder \
     48$ ../source-builder/sb-set-builder \
    4649--log=couverture_qemu_log.txt \
    4750--prefix=$HOME/development/4.12 \
     
    5356
    5457= Coverage Analysis in RTEMS Tester =
    55 '''Previous work:''' There had been 2 previous student work for SOCIS in 2014[1] and 2015[2]. The series of patches detailed in their blogs are the starting point of this project. As years of development has taken place since their work, not all the patches still applied. When I pieced them back together, there were build errors in Covoar (the RTEMS coverage reporting tool) and its associated files. In 2015 this setup produced coverage reports for leon3 and pc386, so the first thing to be done  is to revive this setup and generate the Leon3 report again.
     58== Symbol Set Configurtion ==
     59The tests run in rtems-test will be used to check what percentage of the symbols they cover and the symbols to be analysed are in the symbol set config file which can be found here
     60{{{
     61From top directory of rtems-tools
     62$ cd tester/rtems/testing/coverage
     63$ ls
     64Categories.txt    SPARC-Annul-Slot-Explanation.txt  symbol_sets.cfg
     65Explanations.txt  style.css
     66
     67}}}
     68
     69The format of the config file is
     70
     71 symbolset:
     72 name = NAME_OF_SET
     73 lib = ABSOLUTE_PATH_OF_LIBRARY_1
     74 lib = ABSOLUTE_PATH_OF_LIBRARY_2
     75 ..
     76 ..
     77 ..
     78 lib = ABSOLUTE_PATH_OF_LIBRARY_N
     79symbolset:
     80 name = NAME_OF_ANOTHER_SET
     81 lib = ...
     82
     831. Each set is defined with the keyword 'symbolset:'
     842. name = ... defines a suitable descriptive name for the collections of libraries.
     853. lib = ... to add a library of symbols to be used.
     86
     87An example of this is format is the score set for Leon 3.
     88
     89{{{
     90symbolset:                                                                     
     91    name=score                                                                 
     92    lib=sparc-rtems4.12/c/leon3/cpukit/score/libscore.a
     93}}}
     94
     95== Supported BSPS ==
     96Leon 3: Use 'rtems-bsp=leon3-qemu' with rtems-test cmd.
     97
     98== Performing Coverage Analysis with rtems-test command ==
     99The coverage analysis is performed by adding the --coverage flag when running the rtems-test cmd and selecting a bsp config file that supports coverage. Also the bsp build directory is now added as a separate option that can be used
     100by the symbol set file so the length of the file names can be kept down in symbol_set.cfg file.
     101
     102{{{
     103// Create a suitable directory to hold coverage reports and associated files.
     104
     105$ cd
     106$ mkdir coverage_test
     107$ cd coverage_test
     108
     109
     110// Now run the rtems-test cmd with --coverage
     111
     112$ $HOME/development/rtems/test/rtems-tools/tester/rtems-test /
     113--rtems-tools=$HOME/development/rtems/4.12 /
     114--log=coverage-analysis.log /
     115--rtems-bsp=leon3-qemu /
     116--coverage /
     117--rtems-builddir=$HOME/development/rtems/leon3 /
     118sparc-rtems4.12/c/leon3/testsuites
     119
     120}}}
     121
     122After the testsuite is run, covoar then performs multiple runs for all tests on each symbol set in symbol_set.cfg. Covoar sections the objdump into seperate functions and uses the trace data from Couverture-Qemu that records trace blocks as 'executed' and 'not executed' to mark branches as 'taken' and 'not taken'. Covoar then creates coverage maps from the results. The maps are then merged together and a report is generated. The report is a html file that can be found in the directory created above coverage_test called report.html.
     123
     124{{{
     125// End of the output of the RTEMS testsuite run with rtems-test
     126...
     127....
     128[565/565] p:545 f:1   u:4   e:0   I:0   B:3   t:5   i:3   | sparc/leon3: tmtimer01.exe
     129Passed:        549
     130Failed:          1
     131User Input:      4
     132Expected Fail:   0
     133Indeterminate:   0
     134Benchmark:       3
     135Timeout:         5
     136Invalid:         3
     137Total:         565
     138Average test time: 0:00:01.612648
     139Testing time     : 0:15:11.146458
     140Linking executables to /home/cpod/coverage_test/leon3/coverage
     141Symlinks made
     142Running covoar for score
     143covoar -C/home/cpod/coverage_test/leon3/coverage/config -S /home/cpod/coverage_test/leon3/coverage/score.symcfg -O /home/cpod/coverage_test/leon3/test/score /home/cpod/coverage_test/leon3/coverage/*.exe
     144Reading symbols sets configuration for symbol set file: /home/cpod/coverage_test/leon3/coverage/score.symcfg
     145Generating symbol file for score
     146Coverage run for score finished successfully.
     147-----------------------------------------------
     148Generating reports
     149Cleaning workspace up
     150Coverage analysis finished. You can find results in /home/cpod/coverage_test/leon3
     151}}}
     152
    56153
    57154= References =
     
    60157
    61158[2] http://socis2015rtems.blogspot.ie/
    62 
    63 
    64