Changes between Version 15 and Version 16 of GCI/Documentation/CoverageAnalysis/Coverage

Timestamp:

12/08/18 20:24:32 (6 years ago)

Author:

shashvat jain

Comment:

--

GCI/Documentation/CoverageAnalysis/Coverage

@@ +1 @@
+= RTEMS Coverage Analysis =
+
+[[TOC(GCI/Documentation/CoverageAnalysis/Coverage , depth=2)]]
+
+
+RTEMS is used in many critical systems.  It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible.  With this goal in mind, we have set out to expand the RTEMS test suite so that 100% of the RTEMS executive is tested. [http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ Current coverage results]. [http://km-for-socis2012rtems.blogspot.com/2012/08/first-task.html Lessons learned from 2012]
+ 
+There are numerous industry and country specific standards for safety including [http://en.wikipedia.org/wiki/DO-178B FAA DO-178B] for flight software in the United States.  There are similar aviation standards in other countries as well as in domains such as medical devices, trains, medical and military applications.  As a free software project, the RTEMS Project may never have a complete set of certification paperwork available for download.  But we would like to ensure that RTEMS meets the technical requirements that are shared across these safety and quality oriented standards.
+
+We encourage members of the community to help out.  If you are in a domain where a safety or certification standard applies, work with us to understand that standard and guide us to providing a polished RTEMS product that helps meets that criteria.   Providing funding to augment tests, test procedures or documentation that would aid you in using RTEMS in your domain.  Once the artifact is merged into the project, it becomes a community asset that will be easier to maintain.  Plus the increased level of testing ensures that submissions to RTEMS do not negatively impact you.  
+
+Be active and help us meet your application domain requirements while improving the product for all!
+= Applying Coverage Analysis to RTEMS =
+
+
+In order to achieve the 100% tested goal, it is important to define what constitutes 100% tested.  A lot of information exists about how to completely test a software application.  In general, the term [http://en.wikipedia.org/wiki/Code_coverage Code Coverage] is used to refer to the analysis that is performed to determine what portions of the software are tested by the test suite and what portions are not tested.  It should be noted that Code Coverage does not prove correctness, only that all code has been tested.  For some background information on Code Coverage Analysis, see [wiki:Developer/Coverage/Theory Coverage Analysis Theory].
+
+Traditionally, Code Coverage Analysis has been performed by instrumenting the source code or object code or by using special hardware to monitor the instructions executed.  The guidelines for the RTEMS code coverage effort were to use existing tools and to avoid altering the code to be analyzed.  This was accomplished by using a processor simulator that provides coverage analysis information.  The information was processed to determine which instructions are executed.  We called this Object Code Coverage and we defined 100% tested to be 100% Object Code Coverage.
+
+In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed us to simplify the problem space (uncovered code) so that the goal was attainable.  Second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner and we wanted 100% tested to be applicable to as many user configurations as possible.  The first profile that we defined encompassed the RTEMS executive and was called the Baseline/-Os/POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] are discussed below.  Initially, we set out to achieve 100% Object Code Coverage of the Baseline/-Os/POSIX Enabled Profile.
+
+An issue that had to be addressed from the very beginning was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge'' and its replacement ''covoar''.  The ''covoar'' program currently supports the formats produced by the TSIM, Skyeye , and Qemu simulators. 
+= How it is Done Now? =
+
+
+The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a set of processor simulators in conjunction with a set of RTEMS Coverage Scripts.  The simulators are configured to save trace or coverage information which is analyzed by ''covoar'' once the run is complete. ''covoar'' merges coverage information for a set of methods of interest.  The setup for running the Coverage procedure to run this is as follows:
+
+ *  make a "base directory" to work in
+ *  cd "base directory"
+ *  check out ''rtems-testing'' from CVS
+ *  cd rtems-testing
+ *  make
+ *  Edit ''rtems-testing/rtems-coverage/VERSIONS-COVERAGE'' to account for your local setup
+ *  cd ..
+ *  check out ''rtems'' from CVS
+ *  cd rtems/testsuites
+ *  run ..../rtems-testing/rtems-coverage/remove_managers_not_wanted
+  *  This step is necessary to ensure that every symbol has a unique implementation across the entire set of executables.  The "managers not wanted" code places stubs in the executables.
+ *  cd ..
+ *  ./bootstrap
+  *  watch for errors in case removing the managers note wanted script failed.
+ *  make the directory "tarballs" in the base directory.  Output is saved here
+
+Once this is done, actually running coverage is simple.  You have three scripts:
+
+ *  do_coverage - lowest level script. This one takes a number of arguments and knows the standard RTEMS configurations we test.  Use ''do_coverage -?'' for details.  It is common to use this manually when doing partial runs or you are interested in a single configuration.
+ *  run_coverage - one logical level higher.  Has more advanced commands:
+  *  update - updates the source and bootstraps
+  *  BSP_O[sS2][pP][dD] - run a specific configuration for BSP.  The codes match those of the standard runs.
+  *  BSP_baseline - run all 8 standard configurations for BSP.  This actually builds the BSP 4 times.
+ *  coverage_cron - runs update followed by baseline for all supported BSPs.
+
+When you run one, be sure about the following:
+
+ *  The RTEMS toolset must be at the head of your PATH just like a normal build.
+ *  Add ''rtems-testing/bin'' to your PATH.
+
+The output produced by ''covoar'' is actually a set of HTML and simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code. See 
+http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ and drill down to a single run to see the current output.
+
+There is one other script of interest.  If you collect the output tarballs into a single directory, then you can use ''generate_coverage_html'' to generate the HTML which you see before you drill down.
+= Historical: How it was Done Initially =
+
+[[Image(CoverageFlow.png)]]
+
+The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS Coverage Scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the program ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
+
+One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge''.  The ''covmerge'' program currently supports the formats produced by the TSIM and Skyeye simulators.
+<br style="clear: both" />
+
+The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
+
+||= File Name =||= Purpose of File =||
+|| configuration.txt || Details the settings for the coverage run ||
+|| summary.txt || Provides a summary of the results of the coverage run ||
+|| sizes.txt || Provides a list identifying the file name and source line number of each uncovered range along with its size in bytes ||
+|| report.txt || Details the settings for the coverage run ||
+|| Explanations.txt.!NotFound || Contains the Explanations that were not found for this coverage run (see [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] for more information about how and why to use Explanations) ||
+|| annotated.dmp || Provides the disassembled listing of hello.exe with indications of the object code that was not executed ||
+|| hello.num || The symbol table of hello.exe ||
+
+You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyze is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
 
 = What was Discovered =