Changes between Version 18 and Version 19 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Sep 5, 2009, 12:23:33 AM (10 years ago)
Author:
GlennHumphrey
Comment:

Moving information to new page

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v18 v19  
    66
    77
    8 [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory]
    9 
    108RTEMS is used in many critical systems.  It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible.  In this light, we want to ensure that as close to 100% of the generated assembly code is executed by the RTEMS test suite.  We perform automated coverage testing using a processor simulator in conjunction with a set of RTEMS specific support scripts.
    119
     
    1412There are numerous industry and country specific standards for safety including [http://en.wikipedia.org/wiki/DO-178B FAA DO-178B] for flight software in the United States. There are similar aviation standards in other countries as well as in domains such as medical devices, trains, medical and military applications.  As a free software project, the RTEMS Project will never have a complete set of certification paperwork available for download.  But we would like to ensure that RTEMS meets the technical requirements that are shared across these safety and quality oriented standards.
    1513
     14For some background information on Coverage Analysis, see [wiki:Developer/Coverage/CoverageAnalysisTheory Coverage Analysis Theory].
     15
    1616We encourage members of the community to help out.  If you are in a domain where a safety or certification standard applies, work with us to understand that standard and guide us to providing a polished RTEMS product that helps meets that criteria.   Providing funding to augment tests, test procedures or documentation that would aid you in using RTEMS in your domain.  Once the artifact is merged into the project, it becomes a community asset that will be easier to maintain.  Plus the increased level of testing ensures that submissions to RTEMS do not negatively impact you. 
    1717
    1818Be active and help us meet your application domain requirements while improving the product for all!
    19 = Test Theory =
     19= Applying Coverage Analysis to RTEMS =
    2020
    21 
    22 The ultimate goal of the coverage analysis testing is to ensure that every line of generated assembly in a particular configuration is executed by the test suite, that the tests exercise both taking and not taking every branch, and that every line of source code is actually tested. 
    23 
    24 This is beyond statement level coverage in that we have to verify both paths of a C line like the following is executed:
    25 
    26 {{{
    27 x = (y) ? z : a;
    28 }}}
    29 
    30 That is one statement but includes a branch.  That makes it one point to cover when viewed using statement coverage.  But it is two points to cover in our object level coverage of generated assembly.
    3121= Statement Coverage =
    3222
    3323
    34 Statement coverage verifies that each line of source code in a source file is represented by generated assembly and that that assembly code is exercised.  This requires knowing which source files are involved (which we do) and which lines in those files can produce assembly code (which I don't think we do 100%).  We can easily know which lines are comments and blank but beyond that will require some thought. 
     24This requires knowing which source files are involved (which we do) and which lines in those files can produce assembly code (which I don't think we do 100%).  We can easily know which lines are comments and blank but beyond that will require some thought. 
    3525
    3626The current object coverage utility ''covmerge'' can be modified to generate a report of which source lines were covered. It could generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
    3727
    3828This is definitely an open project at this point.
    39 = Full Branch Coverage =
     29= MC/DC =
    4030
    4131
    42 This is also known as [http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage Modified Condition/Decision Coverage].  From the RTEMS testing perspective, this is to verify that every branch instruction in the generated object has been both taken and not taken.  We cannot determine this without help from a simulator or hardware debugger which gathers this information.
     32From the RTEMS testing perspective, this is to verify that every branch instruction in the generated object has been both taken and not taken.  We cannot determine this without help from a simulator or hardware debugger which gathers this information.
    4333= Coding Advice =
    4434