Notice: We have migrated to GitLab launching 2024-05-01 see here: https://gitlab.rtems.org/

Changes between Version 102 and Version 103 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
11/13/18 18:02:18 (6 years ago)
Author:
Marçal Comajoan Cara
Comment:

fix format

Legend:

Unmodified
Added
Removed
Modified

  • TBR/UserManual/RTEMS_Coverage_Analysis

    v102 v103  
    11= RTEMS Coverage Analysis =
    2 
    3 
    42
    53[[TOC(TBR/UserManual/RTEMS_Coverage_Analysis, depth=2)]]
    … …  
    6462= Historical: How it was Done Initially =
    6563
    66 [[Image(CoverageFlow.png)]]]]
     64[[Image(CoverageFlow.png)]]
    6765
    6866The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS Coverage Scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the program ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
    … …  
    7371The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
    7472
    75 {| border="1" style="margin: 1em auto 1em auto;text-align: left;"
    76 |+
    77 |-
    78 |'''File Name''' || '''Purpose of File'''
    79 |-
    80 | configuration.txt || Details the settings for the coverage run
    81 |-
    82 | summary.txt || Provides a summary of the results of the coverage run
    83 |-
    84 | sizes.txt || Provides a list identifying the file name and source line number of each uncovered range along with its size in bytes
    85 |-
    86 | report.txt || Provides the details of each uncovered range
    87 |-
    88 | Explanations.txt.NotFound || Contains the Explanations that were not found for this coverage run (see [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] for more information about how and why to use Explanations)
    89 |-
    90 | annotated.dmp || Provides the disassembled listing of hello.exe with indications of the object code that was not executed
    91 |-
    92 | hello.num || The symbol table of hello.exe
    93 |}
     73||= File Name =||= Purpose of File =||
     74|| configuration.txt || Details the settings for the coverage run ||
     75|| summary.txt || Provides a summary of the results of the coverage run ||
     76|| sizes.txt || Provides a list identifying the file name and source line number of each uncovered range along with its size in bytes ||
     77|| report.txt || Details the settings for the coverage run ||
     78|| Explanations.txt.NotFound || Contains the Explanations that were not found for this coverage run (see [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] for more information about how and why to use Explanations) ||
     79|| annotated.dmp || Provides the disassembled listing of hello.exe with indications of the object code that was not executed ||
     80|| hello.num || The symbol table of hello.exe ||
    9481
    9582You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyze is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
    … …  
    133120When we started the RTEMS Code Coverage effort, we did not immediately capture results to measure progress.  This actually ended up being the correct thing to do since the ''covmerge'' tool was in development and often produced results that were not directly comparable.  Now that the development of ''covmerge'' has largely settled, we can perform coverage runs on several RTEMS release points and see the progress of the coverage effort.  The results shown below were of the Baseline/-Os/POSIX Enabled Profile run on the SPARC/ERC32.
    134121
    135 {| border="1" style="margin: 1em auto 1em auto;text-align: left;"
    136 |+
    137 |-
    138 |'''Release''' || '''Covered %''' || '''Uncovered Ranges''' || '''Uncovered Bytes''' || '''Total Bytes'''
    139 |-
    140 | 4.7 || 77.51 || 454 || 17508 || 77840
    141 |-
    142 | 4.8 || 76.37 || 538 || 21772 || 92140
    143 |-
    144 | 4.9 || 96.41 || 167 || 2532 || 70564
    145 |-
    146 | 4.10 (head 09/09/2009) || 100 || 0 || 0 || 70480
    147 |}
     122||= Release =||= Covered % =||= Uncovered Ranges =||= Uncovered Bytes =||= Total Bytes =||
     123|| 4.7 || 77.51 || 454 || 17508 || 77840 ||
     124|| 4.8 || 76.37 || 538 || 21772 || 92140 ||
     125|| 4.9 || 96.41 || 167 || 2532 || 70564 ||
     126|| 4.10 (head 09/09/2009) || 100 || 0 || 0 || 70480 ||
    148127
    149128Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
    … …  
    276255Initial analysis has been done at -Os which instructs gcc to generate smaller object code.  At -O2 which optimizes for speed, more code is generated and it is often clear looking at the -O2 reports, that there are test cases needed which are not required at -Os.
    277256= References =
    278 
    279 =  ==General Coverage Testing===
    280 
    281 
     257=== General Coverage Testing
    282258 *  [http://en.wikipedia.org/wiki/Code_coverage Code Coverage Definition]
    283259 *  [http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage Modified Condition/Decision Coverage Definition]
    284260 *  [http://googletesting.blogspot.com/2008/03/tott-understanding-your-coverage-data.html TotT: Understanding Your Coverage Data]
    285261
    286 ===Standards and Certifications===
    287 
    288 
     262=== Standards and Certifications
    289263 *  FAA DO-178B - United States Aviation Standard