Changes between Version 92 and Version 93 of TBR/UserManual/RTEMS_Coverage_Analysis


Ignore:
Timestamp:
Mar 12, 2011, 8:50:04 PM (9 years ago)
Author:
JoelSherrill
Comment:

/* How it was Done */ Add covoar and new way

Legend:

Unmodified
Added
Removed
Modified
  • TBR/UserManual/RTEMS_Coverage_Analysis

    v92 v93  
    2121
    2222In addition to defining the method for determining 100% tested, it is also important to define what is actually being tested.  We accomplished this by defining a set of [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] that allowed us to specify the feature set, configuration options and compiler options used when performing the analysis.  This was important for two reasons.  First, it allowed us to simplify the problem space (uncovered code) so that the goal was attainable.  Second, we wanted to recognize that not all RTEMS users configure RTEMS in the same manner and we wanted 100% tested to be applicable to as many user configurations as possible.  The first profile that we defined encompassed the RTEMS executive and was called the Baseline/-Os/POSIX Enabled Profile.  The RTEMS executive is a large body of code that is generally defined to contain the score, sapi, rtems, and posix directories in the cpukit directory.  This represents the full tasking and synchronization feature set.  More details about [wiki:RTEMS_Coverage_Analysis#Coverage_Profiles Coverage Profiles] are discussed below.  Initially, we set out to achieve 100% Object Code Coverage of the Baseline/-Os/POSIX Enabled Profile.
    23 = How it was Done =
     23
     24An issue that had to be addressed from the very beginning was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge'' and its replacement ''covoar''.  The ''covoar'' program currently supports the formats produced by the TSIM, Skyeye , and Qemu simulators.
     25= How it is Done Now? =
     26
     27
     28The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a set of processor simulators in conjunction with a set of RTEMS Coverage Scripts.  The simulators are configured to save trace or coverage information which is analysed by ''covoar'' once the run is complete. ''covoar'' merges coverage information for a set of methods of interest.  The setup for running the Coverage procedure to run this is as follows:
     29
     30 *  make a "base directory" to work in
     31 *  cd "base directory"
     32 *  check out ''rtems-testing'' from CVS
     33 *  cd rtems-testing
     34 *  make
     35 *  Edit ''rtems-testing/rtems-coverage/VERSIONS-COVERAGE'' to account for your local setup
     36 *  cd ..
     37 *  check out ''rtems'' from CVS
     38 *  cd rtems/testsuites
     39 *  run ..../rtems-testing/rtems-coverage/remove_managers_not_wanted
     40  *  This step is necessary to ensure that every symbol has a unique implementation across the entire set of executables.  The "managers not wanted" code places stubs in the executables.
     41 *  cd ..
     42 *  ./bootstrap
     43  *  watch for errors in case removing the managers note wanted script failed.
     44 *  make the directory "tarballs" in the base directory.  Output is saved here
     45
     46Once this is done, actually running coverage is simple.  You have three scripts:
     47
     48 *  do_coverage - lowest level script. This one takes a number of arguments and knows the standard RTEMS configurations we test.  Use ''do_coverage -?'' for details.  It is common to use this manually when doing partial runs or you are interested in a single configuration.
     49 *  run_coverage - one logical level higher.  Has more advanced commands:
     50  *  update - updates the source and bootstraps
     51  *  BSP_O[sS2][pP][dD] - run a specific configuration for BSP.  The codes match those of the standard runs.
     52  *  BSP_baseline - run all 8 standard configurations for BSP.  This actually builds the BSP 4 times.
     53 *  coverage_cron - runs update followed by baseline for all supported BSPs.
     54
     55When you run one, be sure about the following:
     56
     57 *  The RTEMS toolset must be at the head of your PATH just like a normal build
     58 *  Add ''rtems-testing/bin'' to your PATH
     59
     60The output produced by ''covoar'' is actually a set of HTML and simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code. See
     61http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ and drill down to a single run to see the current output.
     62
     63There is one other script of interest.  If you collect the output tarballs into a single directory, then you can use ''generate_coverage_html'' to generate a the HTML which you see before you drill down.
     64= Historical: How it was Done Initially =
    2465
    2566[[Image(CoverageFlow.png)]]]]
     
    5293|}
    5394
    54 You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyze is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
     95You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyse is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
    5596= What was Discovered =
    5697
     
    184225= References =
    185226
    186 = General Coverage Testing =
     227=  ==General Coverage Testing===
    187228
    188229
     
    190231 *  [http://en.wikipedia.org/wiki/Modified_Condition/Decision_Coverage Modified Condition/Decision Coverage Definition]
    191232 *  [http://googletesting.blogspot.com/2008/03/tott-understanding-your-coverage-data.html TotT: Understanding Your Coverage Data]
    192 =  ==Standards and Certifications===
     233
     234===Standards and Certifications===
    193235
    194236