Changes between Version 42 and Version 43 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Sep 16, 2009, 8:44:31 PM (10 years ago)
Author:
GlennHumphrey
Comment:

Added a new section for information about measuring progress

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v42 v43  
    5656
    5757
     58When we began the RTEMS Code Coverage effort, we performed coverage analysis on the developmental head of RTEMS 4.8 using the POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  We also observed some interesting things about the code we were analyzing.  We noticed that the use of inlining sometimes caused significant branch explosion.  This generated a lot of uncovered ranges that really mapped back to the same source code.  We also found that coding habits and coding style generated a lot of unreachable object code.  Also, the use of a case statement instead of an if statement sometimes lead to unreachable code.
     59
     60Some of our observations were related to the performance of the ''covmerge'' tool.  Of particular interest was the handling of NOP instructions.  Compilers can use NOP instructions to force alignment between functions.  Of course the NOP instructions are not executed and thus had a negative impact on the coverage.  The first attempt at dealing with NOP instructions was to mark them as EXECUTED.  This produced an unwanted side affect of occasionally spliting an uncovered range into two ranges.  We finally settled on an improved method for dealing with NOPs where NOPs were marked as EXECUTED unless they were between two un-EXECUTED instructions.
     61= Resolving Uncovered Code =
     62
     63
     64The coverage analysis provides a report on the ranges of assembly instructions within RTEMS subsystems which are not currently exercised by the tests in the current configuration. Each case has to be individually analysed and addressed.  Historically, we have identified multiple categories for code being uncovered:
     65
     66 *  Needs a new test case
     67
     68 *  Unreachable in current RTEMS configuration.  For example, the SuperCore could have a feature only exercised by a POSIX API object.  It could be disabled when POSIX is not configured.
     69
     70 *  Debug or sanity checking code which can be placed inside an RTEMS_DEBUG conditional.
     71
     72 *  Unreachable paths generated by gcc for switches.  Sometimes you have to restructure switches to avoid unreachable object code.
     73
     74 *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
     75= Measuring Progress =
     76
     77
    5878As mentioned above, the ''covmerge'' script produces reports that contain several metrics used to measure progress.  The first is the number of uncovered object code ranges.  The second is the percent of untested object code as a percentage of the total object code size under analysis.  Together the metrics provide useful information about the status or progress of the Object Code Coverage.
    59 
    60 When we began the RTEMS Code Coverage effort, we performed coverage analysis on the developmental head of RTEMS 4.6 using the POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  Another surprise was that coding habits and style generated a lot of unreachable object code.
    61 = Resolving Uncovered Code =
    62 
    63 
    64 The coverage analysis provides a report on the ranges of assembly instructions within RTEMS subsystems which are not currently exercised by the tests in the current configuration. Each case has to be individually analysed and addressed.  Historically, we have identified multiple categories for code being uncovered:
    65 
    66  *  Needs a new test case
    67 
    68  *  Unreachable in current RTEMS configuration.  For example, the SuperCore could have a feature only exercised by a POSIX API object.  It could be disabled when POSIX is not configured.
    69 
    70  *  Debug or sanity checking code which can be placed inside an RTEMS_DEBUG conditional.
    71 
    72  *  Unreachable paths generated by gcc for switches.  Sometimes you have to restructure switches to avoid unreachable object code.
    73 
    74  *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
    7579= Coverage Profiles =
    7680
     
    176180
    177181 *  pc386
    178 = SPARC =
     182= =SPARC ==
    179183
    180184
     
    184188 *  LEON2
    185189 *  LEON3
    186 = References =
    187 
    188 =  ==General Coverage Testing===
     190=  =References==
     191
     192
     193===General Coverage Testing===
    189194
    190195