Changes between Version 127 and Version 128 of Developer/OpenProjects


Ignore:
Timestamp:
Aug 5, 2009, 9:11:14 PM (10 years ago)
Author:
JoelSherrill
Comment:

/* Coverage Analysis */ Trim down to a reference to RTEMS Test Coverage.

Legend:

Unmodified
Added
Removed
Modified
  • Developer/OpenProjects

    v127 v128  
    373373
    374374
    375 '''Status:''' [wiki:TBR/User/JoelSherrill JoelSherrill] and [wiki:JenniferAverett JenniferAverett] have done work in this area.
    376 
    377 This task consists of performing automated coverage testing using an open
    378 source simulator.  The [wiki:Developer/Simulators/SkyEye SkyEye] project is currently adding coverage
    379 analysis capabilities per our specifications.  When those are available,
    380 the person(s) undertaking this project could analysis the binary object
    381 level coverage provided by the RTEMS Test Suites on any target BSP
    382 supported by the [wiki:Developer/Simulators/SkyEye SkyEye]/RTEMS combination.
    383 
    384 The analysis will identify a subset of RTEMS such as the SuperCore and
    385 a single API implementation and use that as the basis for analysis. RTEMS
    386 includes a lot of source code and the coverage analysis should only focus
    387 on improving the test coverage of that code subset.
    388 
    389 The resulting analysis is expected to provide a report on individual
    390 assembly instructions within RTEMS subsystems which are not currently
    391 exercised by existing tests. Each case has to be individually analyzed
    392 and addressed.  Historically, we have identified multiple categories
    393 for code being uncovered:
    394 
    395  *  Needs a new test case
    396 
    397  *  Unreachable in current RTEMS configuration.  For example, the SuperCore could have a feature only exercised by a POSIX API object.  It could be disabled when POSIX is not configured.
    398 
    399  *  Debug or sanity checking code which can be placed inside an RTEMS_DEBUG conditional.
    400 
    401  *  Unreachable paths generated by gcc for switches.  Sometimes you have to restructure switches to avoid unreachable object code.
    402 
    403  *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, there is often no external way to know the case in question has been hit and no way to do it in a one shot test.
    404 
    405 There are multiple ways to measure progress on this task. In the past, we have used two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
    406 
    407 
    408 This is being taken up as a Google Summer of Code project by Santosh G Vattam and the updates about this project can be found [http://www.rtems.org/wiki/index.php/Coverage_Analysis here]
     375'''Status:''' On going effort.
     376
     377See [wiki:RTEMS_Test_Coverage RTEMS Test Coverage] for current status, ideas, and other information.
     378
     379This is being taken up as a Google Summer of Code 2009 project by Santosh G Vattam and the updates about this project can be found [http://www.rtems.org/wiki/index.php/Coverage_Analysis here]
    409380= Add Sixty-Four Bit Timestamps =
    410381