Changes between Version 33 and Version 34 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
09/15/09 19:21:50 (15 years ago)
Author:
GlennHumphrey
Comment:

Significant re-organization of sections

Legend:

Unmodified
Added
Removed
Modified

  • TBR/UserManual/RTEMS_Coverage_Analysis

    v33 v34  
    2828
    2929Each source of coverage maps (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The covmerge tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  It currently supports the formats produced by the TSIM and Skyeye simulators.
    30 
    3130<br style="clear: both" />
    3231= What was Discovered =
    … …  
    3433
    3534There are multiple ways to measure progress on this task. We primarily use two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
    36 = Beyond Object Code Coverage =
    37 
    38 = Statement Coverage =
    39 
    40 
    41 This requires knowing which source files are involved (which we do) and which lines in those files can produce assembly code (which I don't think we do 100%).  We can easily know which lines are comments and blank but beyond that will require some thought. 
    42 
    43 The current object coverage utility ''covmerge'' can be modified to generate a report of which source lines were covered. It could generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
    44 
    45 This is definitely an open project at this point.
    46 = MC/DC =
    47 
    48 
    49 From the RTEMS testing perspective, this is to verify that every branch instruction in the generated object has been both taken and not taken.  We cannot determine this without help from a simulator or hardware debugger which gathers this information.
    50 
    51 QEMU -- project to do MC/DC .. update here
    52 = Coding Advice =
    53 
    54 = Reasons Code is Not Executed =
     35= Resolving Uncovered Code =
    5536
    5637
    … …  
    6647
    6748 *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
    68 = Impact of Optimization Level =
    69 
    70 
    71 Discuss impact of -O2 versus -Os with example from code.
    72 = Impact of RTEMS Configuration Options =
    73 
    74 
    75 Inlining _Thread_Dispatch_enable, etc.
    76 = Test Procedure =
    77 
    78 
    79 The scripts, tools, and patches are currently in the CVS module gcc-testing in the subdirectory rtems-coverage in the RTEMS CVS Repository.
    80 = Compilation and Configuration Options =
    81 
    82 
    83 TBD
    8449= Coverage Profiles =
    8550
    … …  
    8853
    8954As other support libraries in cpukit is covered, these will be move from the Developmental Profile and added to the '''POSIX Enabled''' and '''Classic API Only''' profiles.
     55= Compilation and Configuration Options =
     56
     57
     58Discuss impact of -O2 versus -Os with example from code.
     59
     60Inlining _Thread_Dispatch_enable, etc.
    9061= POSIX Enabled =
    9162
    … …  
    11485
    11586We welcome your contributions.
     87= Beyond Object Code Coverage =
     88
     89= Statement Coverage =
     90
     91
     92This requires knowing which source files are involved (which we do) and which lines in those files can produce assembly code (which I don't think we do 100%).  We can easily know which lines are comments and blank but beyond that will require some thought. 
     93
     94The current object coverage utility ''covmerge'' can be modified to generate a report of which source lines were covered. It could generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
     95
     96This is definitely an open project at this point.
     97= Condition/Decision Coverage =
     98
     99
     100TBD
     101= MC/DC =
     102
     103
     104From the RTEMS testing perspective, this is to verify that every branch instruction in the generated object has been both taken and not taken.  We cannot determine this without help from a simulator or hardware debugger which gathers this information.
     105
     106QEMU -- project to do MC/DC .. update here
     107= Test Procedure =
     108
     109
     110The scripts, tools, and patches are currently in the CVS module gcc-testing in the subdirectory rtems-coverage in the RTEMS CVS Repository.
    116111= BSPs Analyzed =
    117112