Changes between Version 34 and Version 35 of TBR/UserManual/RTEMS_Coverage_Analysis

Timestamp:

09/15/09 20:34:11 (15 years ago)

Author:

GlennHumphrey

Comment:

Simple edits

TBR/UserManual/RTEMS_Coverage_Analysis

@@ -16 +16 @@
 
 
-In order to achieve the 100% tested goal, it is important to define what constitutes 100% tested.  A lot of information exists about how to completely test a software application.  In general, the term [http://en.wikipedia.org/wiki/Code_coverage Code Coverage] is used to refer to the analysis that is performed to determine what portions of the software are tested by the test suite and what portions are not tested.  For some background information on Code Coverage Analysis, see [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory].
+In order to achieve the 100% tested goal, it is important to define what constitutes 100% tested.  A lot of information exists about how to completely test a software application.  In general, the term [http://en.wikipedia.org/wiki/Code_coverage Code Coverage] is used to refer to the analysis that is performed to determine what portions of the software are tested by the test suite and what portions are not tested.  It should be noted that Code Coverage does not prove correctness only that all code has been tested.  For some background information on Code Coverage Analysis, see [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory].
 
-Traditionally, Code Coverage Analysis has been performed by instrumenting the source code or object code or by using special hardware to monitor the instructions executed.  The guidelines for the RTEMS code coverage effort were to use existing tools and to avoid altering the code to be analyzed.  This was accomplished by using a processor simulator that provides coverage analysis information.  The information was processed to determine which instructions are executed.  We called this object code coverage and we defined 100% tested to be 100% object code coverage.
+Traditionally, Code Coverage Analysis has been performed by instrumenting the source code or object code or by using special hardware to monitor the instructions executed.  The guidelines for the RTEMS code coverage effort were to use existing tools and to avoid altering the code to be analyzed.  This was accomplished by using a processor simulator that provides coverage analysis information.  The information was processed to determine which instructions are executed.  We called this Object Code Coverage and we defined 100% tested to be 100% Object Code Coverage.
 
-In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of Coverage Profiles that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed to simplify the problem space (uncovered code) so that the goal was attainable and second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner.  The first profile that we defined encompassed the RTEMS executive and was called the POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about Coverage Profiles is discussed below.  Initially, we set out to achieve 100% object code coverage of the POSIX Enabled Profile.
+In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of Coverage Profiles that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed us to simplify the problem space (uncovered code) so that the goal was attainable.  Second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner and we wanted 100% tested to be applicable to as many user configurations as possible.  The first profile that we defined encompassed the RTEMS executive and was called the POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about Coverage Profiles are discussed below.  Initially, we set out to achieve 100% Object Code Coverage of the POSIX Enabled Profile.
 = How it was Done =
 
 [[Image(CoverageFlow.png)]]]]
 
-The RTEMS Code Coverage Analysis process was designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS specific support scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the support script ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
+The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS specific support scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the support script ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
 
-Each source of coverage maps (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The covmerge tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  It currently supports the formats produced by the TSIM and Skyeye simulators.
+One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the basic format used by ''covmerge''.  The ''covmerge'' script currently supports the formats produced by the TSIM and Skyeye simulators.
 <br style="clear: both" />
 = What was Discovered =
 
 
-There are multiple ways to measure progress on this task. We primarily use two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
+As mentioned above, the covmerge script produces reports that contain several metrics used to measure progress.  The first is the number of uncovered object code ranges.  The second is the percent of untested object code as a percentage of the total object code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
 = Resolving Uncovered Code =