Changes between Version 79 and Version 80 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Sep 22, 2009, 11:30:57 PM (10 years ago)
Author:
GlennHumphrey
Comment:

Changed references to the Baseline Profile

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v79 v80  
    2020Traditionally, Code Coverage Analysis has been performed by instrumenting the source code or object code or by using special hardware to monitor the instructions executed.  The guidelines for the RTEMS code coverage effort were to use existing tools and to avoid altering the code to be analyzed.  This was accomplished by using a processor simulator that provides coverage analysis information.  The information was processed to determine which instructions are executed.  We called this Object Code Coverage and we defined 100% tested to be 100% Object Code Coverage.
    2121
    22 In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed us to simplify the problem space (uncovered code) so that the goal was attainable.  Second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner and we wanted 100% tested to be applicable to as many user configurations as possible.  The first profile that we defined encompassed the RTEMS executive and was called the POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] are discussed below.  Initially, we set out to achieve 100% Object Code Coverage of the POSIX Enabled Profile.
     22In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed us to simplify the problem space (uncovered code) so that the goal was attainable.  Second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner and we wanted 100% tested to be applicable to as many user configurations as possible.  The first profile that we defined encompassed the RTEMS executive and was called the Baseline/-Os/POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] are discussed below.  Initially, we set out to achieve 100% Object Code Coverage of the Baseline/-Os/POSIX Enabled Profile.
    2323= How it was Done =
    2424
     
    5656
    5757
    58 When we began the RTEMS Code Coverage effort, we performed coverage analysis on the development head of RTEMS 4.8 using the POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  We also observed some interesting things about the code we were analyzing.  We noticed that the use of inlining sometimes caused significant branch explosion.  This generated a lot of uncovered ranges that really mapped back to the same source code.  We also found that some defensive coding habits and coding style idioms could generate unreachable object code.  Also, the use of a case statement that includes all values of an enumerated type instead of an if statement sometimes lead to unreachable code.
     58When we began the RTEMS Code Coverage effort, we performed coverage analysis on the development head of RTEMS 4.8 using the Baseline/-Os/POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  We also observed some interesting things about the code we were analyzing.  We noticed that the use of inlining sometimes caused significant branch explosion.  This generated a lot of uncovered ranges that really mapped back to the same source code.  We also found that some defensive coding habits and coding style idioms could generate unreachable object code.  Also, the use of a case statement that includes all values of an enumerated type instead of an if statement sometimes lead to unreachable code.
    5959
    6060Other observations were related to the performance of the ''covmerge'' tool.  Of particular interest was the handling of NOP instructions.  Compilers can use NOP instructions to force alignment between functions or to fill delay-slots required by the processor.  Of course the NOP instructions are not executed and thus had a negative impact on the coverage.  The first attempt at dealing with NOP instructions was to mark them all as EXECUTED.  This was correct for the NOPs used for function alignment, but not for NOPs used for delay-slots.  Marking delay-slot NOPs as EXECUTED produced an unwanted side effect of occasionally spliting an uncovered range into two ranges.  We finally settled on an improved method for dealing with NOPs where NOPs were marked as EXECUTED unless they were between two NOT EXECUTED instructions.  An example is shown below:
     
    9090As mentioned above, the ''covmerge'' program produces reports that contain several metrics that can be used to measure progress.  The first is the number of uncovered object code ranges.  The second is the percent of untested object code as a percentage of the total object code size under analysis.  Together these metrics provide useful information about the status or progress of the Object Code Coverage.
    9191
    92 When we started the RTEMS Code Coverage effort, we did not immediately capture results to measure progress.  This actually ended up being the correct thing to do since the ''covmerge'' tool was in development and often produced results that were not directly comparable.  Now that the development of ''covmerge'' has largely settled, we can perform coverage runs on several RTEMS release points and see the progress of the coverage effort.  The results shown below were of the POSIX Enabled Profile run on the SPARC/ERC32.
     92When we started the RTEMS Code Coverage effort, we did not immediately capture results to measure progress.  This actually ended up being the correct thing to do since the ''covmerge'' tool was in development and often produced results that were not directly comparable.  Now that the development of ''covmerge'' has largely settled, we can perform coverage runs on several RTEMS release points and see the progress of the coverage effort.  The results shown below were of the Baseline/-Os/POSIX Enabled Profile run on the SPARC/ERC32.
    9393
    9494{| border="1" style="margin: 1em auto 1em auto;text-align: left;"
     
    108108Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
    109109
    110 Now that we have achieved 100% Code Coverage using the POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
     110Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
    111111= Coverage Profiles =
    112112
     
    118118Within the two groups, we recognized the need to use different compiler optimization levels and to analyze each group with POSIX threads enabled and POSIX threads disabled.  Applying these options produced eight sub-groups that we called profiles.  The eight profiles are:
    119119
    120  *  Baseline (-Os, POSIX Enabled)
    121  *  Baseline (-O2, POSIX Enabled)
    122  *  Baseline (-Os, POSIX Disabled)
    123  *  Baseline (-O2, POSIX Disabled)
    124  *  Developmental (-Os, POSIX Enabled)
    125  *  Developmental (-O2, POSIX Enabled)
    126  *  Developmental (-Os, POSIX Disabled)
    127  *  Developmental (-O2, POSIX Disabled)
     120 *  Baseline/-Os/POSIX Enabled
     121 *  Baseline/-O2/POSIX Enabled
     122 *  Baseline/-Os/POSIX Disabled
     123 *  Baseline/-O2/POSIX Disabled
     124 *  Developmental/-Os/POSIX Enabled
     125 *  Developmental/-O2/POSIX Enabled
     126 *  Developmental/-Os/POSIX Disabled
     127 *  Developmental/-O2/POSIX Disabled
    128128
    129129Over time it is desirable to migrate code from the Developmental group to the Baseline.  As support libraries in cpukit become nearly 100% covered, they will be move from the Developmental group to the Baseline group.  Eventually, the Baseline group should contain all of the RTEMS code and the Developmental group should contain nothing.