Changes between Version 21 and Version 22 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
09/08/09 20:44:22 (15 years ago)
Author:
GlennHumphrey
Comment:

Worked on intro material

Legend:

Unmodified
Added
Removed
Modified

  • TBR/UserManual/RTEMS_Coverage_Analysis

    v21 v22  
    66
    77
    8 RTEMS is used in many critical systems.  It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible.  In this light, we want to ensure that as close to 100% of the generated assembly code is executed by the RTEMS test suite.  We perform automated coverage testing using a processor simulator in conjunction with a set of RTEMS specific support scripts.
    9 
    10 There are multiple ways to measure progress on this task. We primarily use two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
    11 
    12 There are numerous industry and country specific standards for safety including [http://en.wikipedia.org/wiki/DO-178B FAA DO-178B] for flight software in the United States. There are similar aviation standards in other countries as well as in domains such as medical devices, trains, medical and military applications.  As a free software project, the RTEMS Project will never have a complete set of certification paperwork available for download.  But we would like to ensure that RTEMS meets the technical requirements that are shared across these safety and quality oriented standards.
    13 
    14 For some background information on Coverage Analysis, see [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory].
     8RTEMS is used in many critical systems.  It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible.  With this goal in mind, we have set out to expand the RTEMS test suite so that 100% of the RTEMS executive is tested.
     9 
     10There are numerous industry and country specific standards for safety including [http://en.wikipedia.org/wiki/DO-178B FAA DO-178B] for flight software in the United States.  There are similar aviation standards in other countries as well as in domains such as medical devices, trains, medical and military applications.  As a free software project, the RTEMS Project will never have a complete set of certification paperwork available for download.  But we would like to ensure that RTEMS meets the technical requirements that are shared across these safety and quality oriented standards.
    1511
    1612We encourage members of the community to help out.  If you are in a domain where a safety or certification standard applies, work with us to understand that standard and guide us to providing a polished RTEMS product that helps meets that criteria.   Providing funding to augment tests, test procedures or documentation that would aid you in using RTEMS in your domain.  Once the artifact is merged into the project, it becomes a community asset that will be easier to maintain.  Plus the increased level of testing ensures that submissions to RTEMS do not negatively impact you. 
    1713
    1814Be active and help us meet your application domain requirements while improving the product for all!
     15= Achieving the Goal =
     16
     17
     18In order to achieve the 100% tested goal, it is important to define what constitutes 100% tested.  A lot of information exists about how to completely test a software application.  In general, the term Code Coverage is used to refer to the analysis that is performed to determine what portions of the software are tested by the test suite and what portions are not tested.
    1919= Applying Coverage Analysis to RTEMS =
    2020
     21
     22For some background information on Coverage Analysis, see [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory].
     23
     24We perform automated coverage testing using a processor simulator in conjunction with a set of RTEMS specific support scripts.
     25
     26There are multiple ways to measure progress on this task. We primarily use two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
    2127= Statement Coverage =
    2228