Changes between Version 20 and Version 21 of GCI/Documentation/CoverageAnalysis/Coverage


Ignore:
Timestamp:
Dec 9, 2018, 6:42:50 AM (4 months ago)
Author:
shashvat jain
Comment:

fixed some broken links and added the current status

Legend:

Unmodified
Added
Removed
Modified
  • GCI/Documentation/CoverageAnalysis/Coverage

    v20 v21  
    115115
    116116One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge''.  The ''covmerge'' program currently supports the formats produced by the TSIM and Skyeye simulators.
    117 <br style="clear: both" />
    118117
    119118The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
     
    177176Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
    178177
    179 Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
     178Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
     179
    180180= Coverage Profiles =
    181 
    182181
    183182RTEMS contains a lot of source code and although the primary focus of coverage analysis is to achieve 100% coverage of well-defined code subsets, we would also like to increase the amount of source code analyzed.  In order to manage the increase in a systematic manner, we defined two basic groups of source code.  The first group is called Baseline and the second group is called Developmental.  The Baseline group contains the source code that has achieved (or nearly achieved) 100% Object Code Coverage.  The Developmental group contains the source code for which there are very few test cases and therefore very poor coverage.
     
    197196
    198197Over time it is desirable to migrate code from the Developmental group to the Baseline.  As support libraries in cpukit become nearly 100% covered, they will be move from the Developmental group to the Baseline group.  Eventually, the Baseline group should contain all of the RTEMS code and the Developmental group should contain nothing.
     198
    199199= Compilation and Configuration Options =
    200200
    201 
    202 The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
     201The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:/GCI/Documentation/CoverageAnalysis/Coverage#HowitisDoneNow RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
    203202
    204203Enabling or disabling POSIX allows us to analyze the RTEMS code in its two most commonly used threading configurations.  When POSIX is enabled, RTEMS is configured to use POSIX threads and the POSIX tests are built and executed as part of the test suite.  When POSIX is disabled, RTEMS is configured to use Classic RTEMS threads and the POSIX tests are not included in the test suite.
     204
    205205= Internal Compilation and Configuration Options =
    206206
     
    215215
    216216
    217 At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:Developer/Coverage/Theory Coverage Analysis Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
     217At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:/GCI/Documentation/CoverageAnalysis/Coverage#CoverageAnalysisTheory Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
     218
    218219= Achieving Statement Coverage =
    219 
    220220
    221221Achieving Statement Coverage requires knowing which source files are involved (which ''covoar'' does) and which lines in those files can produce assembly code (which I don't think ''covoar'' can).  If any dead source code in RTEMS is detected by the combination of  gcc and Coverity Scan, then we can assume that all source code in RTEMS is represented in the generated executables. 
     
    357357 *  extending GCC to provide the debug information required to let covoar evaluate DC and MC/DC in C programs.
    358358  *  IDEA: If GCC reliably reports that all conditions within a single ''if condition'' have the same line number, then we can use that information as the basis for the analysis.  Did we execute the proper set of cases for all branch instructions associated with a single debug line number.
     359
    359360= Current Status =
    360361
    361 
    362 The [wiki:Developer/Coverage/Status Code Coverage Status] section lists the RTEMS BSPs on which we are performing (or would like to perform) Object Code Coverage.  We would like to continue to grow this list.  If you know of a simulator that includes coverage analysis, please let us know.
    363 
    364 With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit.  The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
     362== Code Coverage Status ==
     363
     364
     365
     366= Not Currently Analysed =
     367
     368= ARM on Qemu =
     369
     370Qemu has support for the ARM but we do not have a BSP for this configuration.  They claim support for the gumstix so that BSP might be close to working on qemu.  This would allow us to have trace coverage and do more advanced analysis.  The following links may be of help:
     371
     372 *  http://docwiki.gumstix.org/Gumstix_emulation_for_QEMU
     373 *  http://www.gumstix.net/wiki/index.php?title=Making_qemu_images
     374
     375= Blackfin on Skyeye =
     376
     377
     378Since [wiki:Developer/Simulators/SkyEye SkyEye] supports this target architecture, we hope to one day get coverage results on the following BSPs:
     379
     380 *  eZKit553
     381= M68K/Coldfire on Skyeye =
     382
     383
     384[wiki:Developer/Simulators/SkyEye SkyEye] supports the Coldfire but is currently unable to run any RTEMS Coldfire BSP.  Work to improve Skyeye's Coldfire support is welcomed.  We look forward to being able to use it to perform coverage testing on the following BSPs.
     385
     386 *  mcf5206elite
     387
     388== MIPS on Qemu ==
     389
     390Qemu has support for the MIPS but we do not have a BSP for this configuration.
     391
     392== SH on Qemu ==
     393
     394Qemu has support for the SuperH but we do not have a BSP for this configuration.
     395
     396== SPARC/LEON2 on Qemu ==
     397
     398
     399The Couverture project (http://libre.adacore.com/libre/tools/coverage/) is working to add the necessary coverage capabilities to that simulator. The source code for this project is available from http://forge.open-do.org/scm/?group_id=8
     400
     401With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit.
     402
     403{{{#!comment
     404The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
     405this link is not working , find a better link.
     406}}}
    365407
    366408If you are interested in writing some simple parameter check error cases, then take a look at the branch taken/not taken coverage reports for the "core configuration". Some of these are a simple matter of adding missing test cases for bad parameter path.  Other cases are more difficult.  So if you run into trouble with the analysis, ask or skip it. A common pattern is this: