Changes between Version 9 and Version 10 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Aug 21, 2009, 2:28:04 AM (10 years ago)
Author:
JoelSherrill
Comment:

Add more content

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v9 v10  
    99
    1010There are multiple ways to measure progress on this task. We primarily use two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
     11
     12There are numerous industry and country specific standards for safety including [http://en.wikipedia.org/wiki/DO-178B FAA DO-178B] for flight software in the United States. There are similar aviation standards in other countries as well as in domains such as medical devices, trains, medical and military applications.  As a free software project, the RTEMS Project will never have a complete set of certification paperwork available for download.  But we would like to ensure that RTEMS meets the technical requirements that are shared across these safety and quality oriented standards.
     13
     14We encourage members of the community to help out.  If you are in a domain where a safety or certification standard applies, work with us to understand that standard and guide us to providing a polished RTEMS product that helps meets that criteria.   Providing funding to augment tests, test procedures or documentation that would aid you in using RTEMS in your domain.  Once the artifact is merged into the project, it becomes a community asset that will be easier to maintain.  Plus the increased level of testing ensures that submissions to RTEMS do not negatively impact you. 
     15
     16Be active and help us meet your application domain requirements while improving the product for all!
    1117= Test Theory =
    1218
    1319
    14 The goal of the coverage analysis testing is to ensure that every line of generated assembly in a particular configuration is executed by the test suite.  This is beyond statement level coverage in that we have to verify both paths of a C line like the following is executed:
     20The ultimate goal of the coverage analysis testing is to ensure that every line of generated assembly in a particular configuration is executed by the test suite, that the tests exercise both taking and not taking every branch, and that every line of source code is actually tested. 
     21
     22This is beyond statement level coverage in that we have to verify both paths of a C line like the following is executed:
    1523
    1624{{{
     
    1826}}}
    1927
    20 That is one statement but includes a branch.  That makes it one point to cover when viewed using statement coverage.  But it is two points to cover in our full coverage of generated assembly.
     28That is one statement but includes a branch.  That makes it one point to cover when viewed using statement coverage.  But it is two points to cover in our object level coverage of generated assembly.
     29= Statement Coverage =
     30
     31
     32Statement coverage verifies that each line of source code in a source file is represented by generated assembly and that that assembly code is exercised.  This requires knowing which source files are involved (which we do) and which lines in those files can produce assembly code (which I don't think we do 100%).  We can easily know which lines are comments and blank but beyond that will require some thought. 
     33
     34The current object coverage utility ''covmerge'' can be modified to generate a report of which source lines were covered. It could generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
     35
     36This is definitely an open project at this point.
    2137= Full Branch Coverage =
    2238
    2339
    24 TBD
     40This is also known as [http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage Modified Condition/Decision Coverage].  From the RTEMS testing perspective, this is to verify that every branch instruction in the generated object has been both taken and not taken.  We cannot determine this without help from a simulator or hardware debugger which gathers this information.
     41= Coding Advice =
     42
    2543= Reasons Code is Not Executed =
    2644
     
    4159
    4260The scripts, tools, and patches are currently in the CVS module gcc-testing in the subdirectory rtems-coverage in the RTEMS CVS Repository.
     61= Compilation and Configuration Options =
     62
     63
     64TBD
    4365= Coverage Profiles =
    4466
     
    84106
    85107Skyeye supports The MCF5206 but is currently unable to run the RTEMS BSP.  Work to improve Skyeye's Coldfire support is welcomed.
    86 = =i386 ==
     108= i386 =
    87109
    88110== pc386 ==
     
    90112
    91113We have identified using Qemu for the information.  This project (http://libre.adacore.com/libre/tools/coverage/) aims to add the necessary capabilities to that simulator.
    92 =  ==SPARC===
     114= =SPARC ==
    93115
    94116
    95117We are using TSIM from Gaisler Research on all SPARC CPUs. 
    96 
    97 ====ERC32====
    98 
    99 ====LEON2====
     118== ERC32 ==
     119=  ===LEON2====
    100120
    101121====LEON3====
     122
     123
     124===References===
     125
     126
     127 *  http://en.wikipedia.org/wiki/Code_coverage
     128 *  http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage