Changes between Version 20 and Version 21 of GCI/Documentation/CoverageAnalysis/Coverage

Timestamp:

12/09/18 06:42:50 (6 years ago)

Author:

shashvat jain

Comment:

fixed some broken links and added the current status

GCI/Documentation/CoverageAnalysis/Coverage

@@ -115 +115 @@
 
 One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge''.  The ''covmerge'' program currently supports the formats produced by the TSIM and Skyeye simulators.
-<br style="clear: both" />
 
 The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
@@ -177 +176 @@
 Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
 
-Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
+Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
+
 = Coverage Profiles =
-
 
 RTEMS contains a lot of source code and although the primary focus of coverage analysis is to achieve 100% coverage of well-defined code subsets, we would also like to increase the amount of source code analyzed.  In order to manage the increase in a systematic manner, we defined two basic groups of source code.  The first group is called Baseline and the second group is called Developmental.  The Baseline group contains the source code that has achieved (or nearly achieved) 100% Object Code Coverage.  The Developmental group contains the source code for which there are very few test cases and therefore very poor coverage.
@@ -197 +196 @@
 
 Over time it is desirable to migrate code from the Developmental group to the Baseline.  As support libraries in cpukit become nearly 100% covered, they will be move from the Developmental group to the Baseline group.  Eventually, the Baseline group should contain all of the RTEMS code and the Developmental group should contain nothing.
+
 = Compilation and Configuration Options =
 
-
-The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
+The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:/GCI/Documentation/CoverageAnalysis/Coverage#HowitisDoneNow RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
 
 Enabling or disabling POSIX allows us to analyze the RTEMS code in its two most commonly used threading configurations.  When POSIX is enabled, RTEMS is configured to use POSIX threads and the POSIX tests are built and executed as part of the test suite.  When POSIX is disabled, RTEMS is configured to use Classic RTEMS threads and the POSIX tests are not included in the test suite.
+
 = Internal Compilation and Configuration Options =
 
@@ -215 +215 @@
 
 
-At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:Developer/Coverage/Theory Coverage Analysis Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
+At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:/GCI/Documentation/CoverageAnalysis/Coverage#CoverageAnalysisTheory Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
+
 = Achieving Statement Coverage =
-
 
 Achieving Statement Coverage requires knowing which source files are involved (which ''covoar'' does) and which lines in those files can produce assembly code (which I don't think ''covoar'' can).  If any dead source code in RTEMS is detected by the combination of  gcc and Coverity Scan, then we can assume that all source code in RTEMS is represented in the generated executables.  
@@ -357 +357 @@
  *  extending GCC to provide the debug information required to let covoar evaluate DC and MC/DC in C programs.
   *  IDEA: If GCC reliably reports that all conditions within a single ''if condition'' have the same line number, then we can use that information as the basis for the analysis.  Did we execute the proper set of cases for all branch instructions associated with a single debug line number.
+
 = Current Status =
 
-
-The [wiki:Developer/Coverage/Status Code Coverage Status] section lists the RTEMS BSPs on which we are performing (or would like to perform) Object Code Coverage.  We would like to continue to grow this list.  If you know of a simulator that includes coverage analysis, please let us know.
-
-With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit.  The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
+== Code Coverage Status ==
+
+
+
+= Not Currently Analysed =
+
+= ARM on Qemu =
+
+Qemu has support for the ARM but we do not have a BSP for this configuration.  They claim support for the gumstix so that BSP might be close to working on qemu.  This would allow us to have trace coverage and do more advanced analysis.  The following links may be of help:
+
+ *  http://docwiki.gumstix.org/Gumstix_emulation_for_QEMU
+ *  http://www.gumstix.net/wiki/index.php?title=Making_qemu_images
+
+= Blackfin on Skyeye =
+
+
+Since [wiki:Developer/Simulators/SkyEye SkyEye] supports this target architecture, we hope to one day get coverage results on the following BSPs:
+
+ *  eZKit553
+= M68K/Coldfire on Skyeye =
+
+
+[wiki:Developer/Simulators/SkyEye SkyEye] supports the Coldfire but is currently unable to run any RTEMS Coldfire BSP.  Work to improve Skyeye's Coldfire support is welcomed.  We look forward to being able to use it to perform coverage testing on the following BSPs.
+
+ *  mcf5206elite
+
+== MIPS on Qemu ==
+
+Qemu has support for the MIPS but we do not have a BSP for this configuration.
+
+== SH on Qemu ==
+
+Qemu has support for the SuperH but we do not have a BSP for this configuration.
+
+== SPARC/LEON2 on Qemu ==
+
+
+The Couverture project (http://libre.adacore.com/libre/tools/coverage/) is working to add the necessary coverage capabilities to that simulator. The source code for this project is available from http://forge.open-do.org/scm/?group_id=8
+
+With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit. 
+
+{{{#!comment
+The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
+this link is not working , find a better link.
+}}}
 
 If you are interested in writing some simple parameter check error cases, then take a look at the branch taken/not taken coverage reports for the "core configuration". Some of these are a simple matter of adding missing test cases for bad parameter path.  Other cases are more difficult.  So if you run into trouble with the analysis, ask or skip it. A common pattern is this: