Changes between Version 1 and Version 2 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Aug 5, 2009, 11:53:27 PM (10 years ago)
Author:
JoelSherrill
Comment:

First attempt at update

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v1 v2  
    22
    33
    4 This task consists of performing automated coverage testing using an open
    5 source simulator.  The [wiki:Developer/Simulators/SkyEye SkyEye] project is currently adding coverage
    6 analysis capabilities per our specifications.  When those are available,
    7 the person(s) undertaking this project could analysis the binary object
    8 level coverage provided by the RTEMS Test Suites on any target BSP
    9 supported by the [wiki:Developer/Simulators/SkyEye SkyEye]/RTEMS combination.
    104
    11 The analysis will identify a subset of RTEMS such as the SuperCore and
    12 a single API implementation and use that as the basis for analysis. RTEMS
    13 includes a lot of source code and the coverage analysis should only focus
    14 on improving the test coverage of that code subset.
     5[[TOC(TBR/UserManual/RTEMS_Coverage_Analysis, depth=2)]]
    156
    16 The resulting analysis is expected to provide a report on individual
    17 assembly instructions within RTEMS subsystems which are not currently
    18 exercised by existing tests. Each case has to be individually analyzed
    19 and addressed.  Historically, we have identified multiple categories
    20 for code being uncovered:
     7
     8RTEMS is used in many critical systems.  It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible.  In this light, we want to ensure that as close to 100% of the generate assembly code is executed by the RTEMS test suite.  We perform automated coverage testing using an processor simulator in conjunction with a set of RTEMS specific support scripts.
     9
     10
     11
     12
     13There are multiple ways to measure progress on this task. In the past, we have used two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
     14= Test Theory =
     15
     16= Reasons Code is Not Executed =
     17
     18
     19The coverage analysis provides a report on the ranges of assembly instructions within RTEMS subsystems which are not currently exercised by the tests in the current configuration. Each case has to be individually analysed and addressed.  Historically, we have identified multiple categories for code being uncovered:
    2120
    2221 *  Needs a new test case
     
    2827 *  Unreachable paths generated by gcc for switches.  Sometimes you have to restructure switches to avoid unreachable object code.
    2928
    30  *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, there is often no external way to know the case in question has been hit and no way to do it in a one shot test.
     29 *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
     30= Test Procedure =
    3131
    32 There are multiple ways to measure progress on this task. In the past, we have used two metrics.  The first is the reduction in the number of uncovered binary code ranges from that identified initially.  The second is the percent of untested binary object code as a percentage of the total code size under analysis.  Together the metrics provide useful information.  Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.
     32= Coverage Profiles =
     33
     34
     35RTEMS includes a lot of source code and the coverage analysis should focus on improving the test coverage of well-defined code subsets.
     36= Classic and POSIX =
     37
     38
     39This is the first profile we tested. This focused on the score, sapi, rtems, and posix directories in the cpukit directory.  This profile represents a full tasking and synchronization feature set.
     40= Classic Only =
     41
     42
     43In this profile, we disabled POSIX and focus on the contents of the score, sapi, and rtems directories in the cpukit directory.  The POSIX API and tests are disabled.  In this profile, we expect to identify:
     44
     45 *  features in score only exercised by POSIX
     46 *  features in score available via Classic API but only tested via POSIX
     47 *  POSIX features like sleep() which are enabled with POSIX threads are disabled.
     48
     49The first case will allow us to disable score features in this configuration and reduce the code size.
     50
     51The second case allows us to approach 100% coverage in every RTEMS configuration.
     52= Classic, POSIX, and Libc Support =
     53
     54
     55This adds the libcsupport subdirectory to the Classic and POSIX Coverage Profile. 
     56
     57We do not test this profile yet.
     58= BSPs Analyzed =
     59
     60= ARM =
     61
     62
     63The [wiki:Developer/Simulators/SkyEye SkyEye] project has added coverage analysis capabilities per our specifications. We are currently using it on some ARM targets to generate coverage reports.
     64== EDB7312 ==
     65== GumStix ==
     66
     67= =SPARC ==
     68
     69
     70We are using TSIM from Gaisler Research on all SPARC CPUs. 
     71=  ===ERC32====
     72
     73====LEON2====
     74
     75====LEON3====