Notice: We have migrated to GitLab launching 2024-05-01 see here: https://gitlab.rtems.org/

Changes between Version 16 and Version 17 of GCI/Documentation/CoverageAnalysis/Coverage


Ignore:
Timestamp:
12/08/18 20:35:03 (5 years ago)
Author:
shashvat jain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • GCI/Documentation/CoverageAnalysis/Coverage

    v16 v17  
    2222An issue that had to be addressed from the very beginning was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge'' and its replacement ''covoar''.  The ''covoar'' program currently supports the formats produced by the TSIM, Skyeye , and Qemu simulators.
    2323= How it is Done Now? =
    24 
    25 
    26 The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a set of processor simulators in conjunction with a set of RTEMS Coverage Scripts.  The simulators are configured to save trace or coverage information which is analyzed by ''covoar'' once the run is complete. ''covoar'' merges coverage information for a set of methods of interest.  The setup for running the Coverage procedure to run this is as follows:
    27 
    28  *  make a "base directory" to work in
    29  *  cd "base directory"
    30  *  check out ''rtems-testing'' from CVS
    31  *  cd rtems-testing
    32  *  make
    33  *  Edit ''rtems-testing/rtems-coverage/VERSIONS-COVERAGE'' to account for your local setup
    34  *  cd ..
    35  *  check out ''rtems'' from CVS
    36  *  cd rtems/testsuites
    37  *  run ..../rtems-testing/rtems-coverage/remove_managers_not_wanted
    38   *  This step is necessary to ensure that every symbol has a unique implementation across the entire set of executables.  The "managers not wanted" code places stubs in the executables.
    39  *  cd ..
    40  *  ./bootstrap
    41   *  watch for errors in case removing the managers note wanted script failed.
    42  *  make the directory "tarballs" in the base directory.  Output is saved here
    43 
    44 Once this is done, actually running coverage is simple.  You have three scripts:
    45 
    46  *  do_coverage - lowest level script. This one takes a number of arguments and knows the standard RTEMS configurations we test.  Use ''do_coverage -?'' for details.  It is common to use this manually when doing partial runs or you are interested in a single configuration.
    47  *  run_coverage - one logical level higher.  Has more advanced commands:
    48   *  update - updates the source and bootstraps
    49   *  BSP_O[sS2][pP][dD] - run a specific configuration for BSP.  The codes match those of the standard runs.
    50   *  BSP_baseline - run all 8 standard configurations for BSP.  This actually builds the BSP 4 times.
    51  *  coverage_cron - runs update followed by baseline for all supported BSPs.
    52 
    53 When you run one, be sure about the following:
    54 
    55  *  The RTEMS toolset must be at the head of your PATH just like a normal build.
    56  *  Add ''rtems-testing/bin'' to your PATH.
    57 
    58 The output produced by ''covoar'' is actually a set of HTML and simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code. See
    59 http://www.rtems.org/ftp/pub/rtems/people/joel/coverage/ and drill down to a single run to see the current output.
    60 
    61 There is one other script of interest.  If you collect the output tarballs into a single directory, then you can use ''generate_coverage_html'' to generate the HTML which you see before you drill down.
    62 = Historical: How it was Done Initially =
    63 
    64 [[Image(CoverageFlow.png)]]
    65 
    66 The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS Coverage Scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the program ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
    67 
    68 One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge''.  The ''covmerge'' program currently supports the formats produced by the TSIM and Skyeye simulators.
    69 <br style="clear: both" />
    70 
    71 The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
    72 
    73 ||= File Name =||= Purpose of File =||
    74 || configuration.txt || Details the settings for the coverage run ||
    75 || summary.txt || Provides a summary of the results of the coverage run ||
    76 || sizes.txt || Provides a list identifying the file name and source line number of each uncovered range along with its size in bytes ||
    77 || report.txt || Details the settings for the coverage run ||
    78 || Explanations.txt.!NotFound || Contains the Explanations that were not found for this coverage run (see [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] for more information about how and why to use Explanations) ||
    79 || annotated.dmp || Provides the disassembled listing of hello.exe with indications of the object code that was not executed ||
    80 || hello.num || The symbol table of hello.exe ||
    81 
    82 You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyze is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
    83 
    84 = What was Discovered =
    85 
    86 
    87 When we began the RTEMS Code Coverage effort, we performed coverage analysis on the development head of RTEMS 4.8 using the Baseline/-Os/POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  We also observed some interesting things about the code we were analyzing.  We noticed that the use of inlining sometimes caused significant branch explosion.  This generated a lot of uncovered ranges that really mapped back to the same source code.  We also found that some defensive coding habits and coding style idioms could generate unreachable object code.  Also, the use of a case statement that includes all values of an enumerated type instead of an if statement sometimes lead to unreachable code.
    88 
    89 Other observations were related to the performance of the ''covmerge'' tool.  Of particular interest was the handling of NOP instructions.  Compilers can use NOP instructions to force alignment between functions or to fill delay-slots required by the processor.  Of course the NOP instructions are not executed and thus had a negative impact on the coverage.  The first attempt at dealing with NOP instructions was to mark them all as EXECUTED.  This was correct for the NOPs used for function alignment, but not for NOPs used for delay-slots.  Marking delay-slot NOPs as EXECUTED produced an unwanted side effect of occasionally spliting an uncovered range into two ranges.  We finally settled on an improved method for dealing with NOPs where NOPs were marked as EXECUTED unless they were between two NOT EXECUTED instructions.  An example is shown below:
    90 
    91 {{{
    92 2003ee8:  80 a6 20 00   cmp  %i0, 0                            <== NOT EXECUTED
    93 2003eec:  02 80 00 06   be  2003f04 <IMFS_fifo_write+0x60>     <== NOT EXECUTED
    94 2003ef0:  01 00 00 00   nop                                    <== NOT EXECUTED
    95 2003ef4:  40 00 78 fb   call  20222e0 <__errno>                <== NOT EXECUTED
    96 2003ef8:  b0 20 00 18   neg  %i0                               <== NOT EXECUTED
    97 }}}
    98 
    99 This solution to the NOP problem was important because NOPs were falsely increasing the number of uncovered ranges.  This created an unnecessary explosion of the reports and increased the uncovered ranges to examine.
    100 = Resolving Uncovered Code =
    101 
    102 
    103 The output files produced by ''covmerge'' are intended to provide both a quick-look at the status of a coverage run and the details needed to resolve the uncovered ranges.  As we worked through the resolution of the uncovered ranges, we noticed that the uncovered ranges usually fit into one of the following categories:
    104 
    105  *  A new test case is needed.
    106 
    107  *  Code unreachable in selected RTEMS configuration.  For example, the !SuperCore could have a feature only exercised by a POSIX API object.  It should be disabled when POSIX is not configured.
    108 
    109  *  Debug or sanity checking code which should be placed inside an RTEMS_DEBUG conditional.
    110 
    111  *  Unreachable paths generated for switch statements.  If the switch is based upon an enumerated type and the switch includes cases for all values, then it must be possible to actually generate all values at this point in the code. You can restructure the switch to only include possible values and thus avoid unreachable object code.  This is sometimes best done by rewriting the switch into a series of if/else statements.
    112 
    113  *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
    114 
    115 In general, it is interesting to note that the resolution of uncovered code does not simply translate into additions to the test suite.  Often the resolution points to improvements or changes to the analyzed code.  This can lead to more intelligent factoring of the code or a code re-design that produces a simpler solution.  There is also the notion that just because the analyzed code is "good" the way it is does not mean that it should not be rewritten to improve its testability.  Code that is completely tested is '''always''' better.
    116 = Measuring Progress =
    117 
    118 
    119 As mentioned above, the ''covmerge'' program produces reports that contain several metrics that can be used to measure progress.  The first is the number of uncovered object code ranges.  The second is the percent of untested object code as a percentage of the total object code size under analysis.  Together these metrics provide useful information about the status or progress of the Object Code Coverage.
    120 
    121 When we started the RTEMS Code Coverage effort, we did not immediately capture results to measure progress.  This actually ended up being the correct thing to do since the ''covmerge'' tool was in development and often produced results that were not directly comparable.  Now that the development of ''covmerge'' has largely settled, we can perform coverage runs on several RTEMS release points and see the progress of the coverage effort.  The results shown below were of the Baseline/-Os/POSIX Enabled Profile run on the SPARC/ERC32.
    122 
    123 ||= Release =||= Covered % =||= Uncovered Ranges =||= Uncovered Bytes =||= Total Bytes =||
    124 || 4.7 || 77.51 || 454 || 17508 || 77840 ||
    125 || 4.8 || 76.37 || 538 || 21772 || 92140 ||
    126 || 4.9 || 96.41 || 167 || 2532 || 70564 ||
    127 || 4.10 (head 09/09/2009) || 100 || 0 || 0 || 70480 ||
    128 
    129 Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
    130 
    131 Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
    132 = Coverage Profiles =
    133 
    134 
    135 RTEMS contains a lot of source code and although the primary focus of coverage analysis is to achieve 100% coverage of well-defined code subsets, we would also like to increase the amount of source code analyzed.  In order to manage the increase in a systematic manner, we defined two basic groups of source code.  The first group is called Baseline and the second group is called Developmental.  The Baseline group contains the source code that has achieved (or nearly achieved) 100% Object Code Coverage.  The Developmental group contains the source code for which there are very few test cases and therefore very poor coverage.
    136 
    137 Initially, the Baseline group included source code from the cpukit.  Specifically the following cpukit directories were included:  score, sapi, rtems and posix.  This group represents a full tasking and synchronization feature set.  What was not in the Baseline group was placed in the Developmental group.  The Developmental group included: libcsupport, libfs/imfs, libmisc/stackchk, libmisc/cpuuse, libmisc/bspcmdline, libmisc/dmpbuf and libmisc/devnull.
    138 
    139 Within the two groups, we recognized the need to use different compiler optimization levels and to analyze each group with POSIX threads enabled and POSIX threads disabled.  Applying these options produced eight sub-groups that we called profiles.  The eight profiles are:
    140 
    141  *  Baseline/-Os/POSIX Enabled
    142  *  Baseline/-O2/POSIX Enabled
    143  *  Baseline/-Os/POSIX Disabled
    144  *  Baseline/-O2/POSIX Disabled
    145  *  Developmental/-Os/POSIX Enabled
    146  *  Developmental/-O2/POSIX Enabled
    147  *  Developmental/-Os/POSIX Disabled
    148  *  Developmental/-O2/POSIX Disabled
    149 
    150 Over time it is desirable to migrate code from the Developmental group to the Baseline.  As support libraries in cpukit become nearly 100% covered, they will be move from the Developmental group to the Baseline group.  Eventually, the Baseline group should contain all of the RTEMS code and the Developmental group should contain nothing.
    151 = Compilation and Configuration Options =
    152 
    153 
    154 The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
    155 
    156 Enabling or disabling POSIX allows us to analyze the RTEMS code in its two most commonly used threading configurations.  When POSIX is enabled, RTEMS is configured to use POSIX threads and the POSIX tests are built and executed as part of the test suite.  When POSIX is disabled, RTEMS is configured to use Classic RTEMS threads and the POSIX tests are not included in the test suite.
    157 = Internal Compilation and Configuration Options =
    158 
    159 
    160 There are several compilation and configuration options that are built into the RTEMS Coverage Scripts and are not selectable from the command line.  These options effect the RTEMS build and are used to simplify the code to aid analysis.  Ideally, we would like the coverage build to match the default build for RTEMS.  Over time, we will work to eliminate the need for the internal options.  The current options being used are:
    161 
    162  *  NDEBUG=1 - Disables asserts.  We will probably keep this option.
    163  *  RTEMS_DO_NOT_INLINE_THREAD_ENABLE_DISPATCH=1 - Inlining resulted in branch explosion.  Over 200 new test cases will be needed to eliminate this option.
    164  *  RTEMS_DO_NOT_INLINE_CORE_MUTEX_SEIZE=1 - Inlining resulted in very difficult code to analyze.  This option should be able to be eliminated.
    165  *  RTEMS_DO_NOT_UNROLL_THREADQ_ENQUEUE_PRIORITY=1 - Unrolling loop resulted in multiple interrupt critical sections.  This option should be able to be eliminated.
    166 = Beyond Object Code Coverage =
    167 
    168 
    169 At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:Developer/Coverage/Theory Coverage Analysis Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
    170 = Achieving Statement Coverage =
    171 
    172 
    173 Achieving Statement Coverage requires knowing which source files are involved (which ''covoar'' does) and which lines in those files can produce assembly code (which I don't think ''covoar'' can).  If any dead source code in RTEMS is detected by the combination of  gcc and Coverity Scan, then we can assume that all source code in RTEMS is represented in the generated executables. 
    174 
    175 The current object coverage utility ''covoar'' reports on which source lines were covered.  It could easily be modified to generate a report indicating which source lines were covered, not covered, or only partially covered.
    176 
    177 ''covoar'' could also generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
    178 
    179 Adding a statement coverage report to ''covoar'' is an open project.
    180 = Achieving !Condition/Decision Coverage =
    181 
    182 
    183 Achieving !Condition/Decision Coverage requires knowing whether each branch has been both taken and not taken.  Currently QEMU and tsim can be used to gather this information.
    184 
    185 tsim produces bitmaps indicating instruction executed, branch taken, and branch not taken.
    186 
    187 All versions of QEMU produce a debug log of the instructions executed when an executable is run.  The trace information is analyzed to identify branch instructions and to determine whether the branch was taken and/or not taken.  Some versions of QEMU may also be able to produce a trace log which is denser but contains the same information.
    188 
    189 skyeye does not produce branch taken/not taken information.
    190 
    191 ''covoar'' produces reports on which branch instructions are taken and not taken.  Our goal is to ensure that each branch instruction is taken and not taken. 
    192 
    193 GCC does not include debug information which indicates that a sequence of compare and branch instructions are part of a single logical condition.  This hinders our ability to augment ''covoar'' to make direct claims regarding Decision Coverage (DC) and Modified condition/decision coverage (MC/DC).
    194 
    195 We believe that for single condition ''if'' statements such as ''if (cond) action'' or ''if (cond) action1 else action2'', that we are achieving full DC and MC/DC coverage because all logical paths are exercised.
    196 
    197 Similarly given a dual OR condition ''if'' statement (in C) such as one the following:
    198 
    199 {{{
    200 Case OR1: if (cond1 or cond2)
    201             action
    202 Case OE2: if (cond1 or cond2)
    203             action1
    204           else
    205             action2
    206 }}}
    207 
    208 We aim for the following cases given our branch coverage requirements:
    209 
    210  *  cond1 branch taken, cond2 short-circuited
    211  *  cond1 branch not taken, cond2 taken
    212  *  cond1 branch not taken, cond2 not taken
    213 
    214 As the above set of test cases represent the entire set of possible execution paths, we have achieved DC and MC/DC level coverage.
    215 
    216 {{{
    217 Case AND1: if (cond1 and cond2)
    218              action
    219 Case AND2: if (cond1 and cond2)
    220              action1
    221            else
    222              action2
    223 }}}
    224 
    225 We aim for the following cases given our branch coverage requirements:
    226 
    227  *  cond1 branch taken, cond2 taken
    228  *  cond1 branch taken, cond2 not taken
    229  *  cond1 branch not taken, cond2 short-circuited
    230 
    231 Again, the above set of test cases represent the entire set of possible execution paths, we have achieved DC and MC/DC level coverage.
    232 
    233 Open projects in this area include:
    234 
    235  *  proving our branch coverage testing policy meets decision coverage (DC) requirements in a more general sense.
    236  *  extending GCC to provide the debug information required to let covoar evaluate DC and MC/DC in C programs.
    237   *  IDEA: If GCC reliably reports that all conditions within a single ''if condition'' have the same line number, then we can use that information as the basis for the analysis.  Did we execute the proper set of cases for all branch instructions associated with a single debug line number.
    238 = Current Status =
    239 
    240 
    241 The [wiki:Developer/Coverage/Status Code Coverage Status] section lists the RTEMS BSPs on which we are performing (or would like to perform) Object Code Coverage.  We would like to continue to grow this list.  If you know of a simulator that includes coverage analysis, please let us know.
    242 
    243 With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit.  The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
    244 
    245 If you are interested in writing some simple parameter check error cases, then take a look at the branch taken/not taken coverage reports for the "core configuration". Some of these are a simple matter of adding missing test cases for bad parameter path.  Other cases are more difficult.  So if you run into trouble with the analysis, ask or skip it. A common pattern is this:
    246 
    247 {{{
    248 if (arg1 bad)
    249   return EINVAL;
    250 if (arg2 bad)
    251   return EINVAL;
    252 }}}
    253 
    254 GCC is smart enough to optimize the returns into one block of code.  Thus we could have a test for arg1 or arg2 bad and obtain 100% instruction coverage.  But would not get 100% branch coverage.
    255 
    256 Initial analysis has been done at -Os which instructs gcc to generate smaller object code.  At -O2 which optimizes for speed, more code is generated and it is often clear looking at the -O2 reports, that there are test cases needed which are not required at -Os.
    257 
    258 == Coverage Analysis Theory ==
    259 
    260 The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intended to summarise or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
    261 
    262 The ultimate goal of Code Coverage Analysis is to ensure that a test suite adequately tests a particular body of code.  In order to achieve this goal, several different coverage criteria may have to be examined.  Let's consider the following criteria:
    263 
    264  *  '''Statement Coverage''' - Has each line of the source code been executed?
    265  *  '''Decision Coverage''' (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
    266  *  '''Condition Coverage''' - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
    267  *  '''Object Coverage''' - Has each line of generated assembly been executed?
    268 
    269 === Statement Coverage ===
    270 
    271 
    272 Statement Coverage requires that each line of source code must be executed.  This is often considered the simplest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for identifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
    273 
    274 === Decision Coverage ===
    275 
    276 
    277 Decision Coverage requires that each control structure evaluate to both TRUE and FALSE.  This is a pretty good criteria because it generally ensures that both the TRUE and FALSE paths of an expression are covered.  However, short-circuit operators will prevent some portions of a complex expression from being evaluated.
    278 === Condition Coverage ===
    279 
    280 
    281 
    282 Condition Coverage requires that each boolean sub-expression evaluate to both TRUE and FALSE.  This criteria goes a little further than Decision Coverage by ensuring that the component parts of a compound expression each evaluate to TRUE and FALSE.  But it should be noted that Condition Coverage by itself does not necessarily imply decision coverage.  Because of this fact, it is best to apply Decision Coverage and Condition Coverage together.
    283 
    284 === Object Coverage ===
    285 
    286 
    287 Object Coverage requires that each line of generated assembly be executed.  This can be a very good general criteria because it ensures most of the test cases that the other criteria ensure.
    288 
    289 === Criteria Relationships ===
    290 
    291 
    292 [[Image(CoverageCategories.png)]]
    293 
    294 Each of these criteria can be used independently to analyze the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completeness and complexity of the test suite increases as it satisfies first Statement Coverage and then Decision Coverage and finally !Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and !Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy !Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and !Condition/Decision Coverage is the most complex to satisfy.
    295 
    296 === An Example ===
    297 
    298 
    299 In order to illustrate what is covered by each of the different criteria, consider the following example showing the source code for a simple if statement along with its generated pseudo-code instructions.
    300 
    301 || '''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code''' ||
    302 || A || if (x OR y) || A1 || cmp x, 0 branch if FALSE to do something ||
    303 || || || A2 || cmp y, 0 branch if TRUE around do something ||
    304 || B || do something || B || do something instructions ||
    305 
    306 ==== Statement Coverage ====
    307 
    308 
    309 A single test case that allows the if statement to evaluate to TRUE will execute blocks A and B.  This will achieve 100% Statement Coverage.
    310 
    311 ==== Decision Coverage ====
    312 
    313 
    314 A minimum of two test cases are required to achieve 100% Decision Coverage.  One case must force the if statement to evaluate to TRUE and the other case must force the if statement to evaluate to FALSE.  A test case that forces a TRUE outcome will either execute blocks A1 and B or A1, A2 and B.  A test case that forces a FALSE outcome will execute blocks A1 and A2.
    315 
    316 ==== !Condition/Decision Coverage ====
    317 
    318 
    319 A minimum of two test cases are required to achieve 100% !Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
    320 
    321 ==== Object Coverage ====
    322 
    323 
    324 One carefully chosen test case where x is FALSE and y is TRUE will achieve 100% Object Coverage.  The test case will execute blocks A1, A2 and B.
    325 
    326 {{{#!comment
    327 TBD : MC/DC Analysis : give a technical description on MC/DC .UNDER THEORY
    328 }}}
    329 
    330 
    331 {{{#!comment
    332 = Improve Coverage Analysis Toolset =
    333 
    334 '''Mentors:''' Chris Johns, Joel Sherrill, C.P. O'Donnell
    335 
    336 '''Students:''' Vijay Kumar Banerjee
    337 
    338 '''Progress:''' The Coverage Analysis is running and generating coverage reports in html and txt format
    339 
    340 '''Blockers:''' Generating .gcno notes files by changing gcc flags
    341 
    342 ''' Development Blog :''' https://thelunatic.github.io/rtems_gsoc18
    343 }}}
    344 
    34524
    34625=== Running covoar To generate coverage reports ===
    … …  
    421100[[Image(sizes_edit.png)]]
    422101
     102
     103{{{#!comment
     104TBD: MC/DC Analysis and other technical stuff in current time .
     105}}}
     106
     107
     108= Historical: How it was Done Initially =
     109
     110[[Image(CoverageFlow.png)]]
     111
     112The RTEMS Code Coverage Analysis process is designed to be as automated as possible.  The coverage testing is performed using a processor simulator in conjunction with a set of RTEMS Coverage Scripts.  The code to be analyzed is linked together as a single relocatable with special start (''start_coverage'') and end (''end_coverage'') symbols.  The relocatable is then linked to the same address in every test from the test suite.  Each test is then executed on a processor simulator that gathers information about which instructions were executed and produces a coverage map for the test.  After all tests have finished, the program ''covmerge'' is used to merge all coverage maps into a unified coverage map for the entire test suite and to produce reports that identify the uncovered code.  The picture shown provides the general flow of the process.
     113
     114One issue that had to be addressed was the different coverage map formats.  Each source of a coverage map (e.g. simulator, hardware debugger, etc.) may produce a coverage map in a different format.  The ''covmerge'' tool is implemented using C++ classes and provides for inheriting new Coverage Reader and Writer classes for specific coverage map formats.  This allows different formats to be converted to the internal representation used by ''covmerge''.  The ''covmerge'' program currently supports the formats produced by the TSIM and Skyeye simulators.
     115<br style="clear: both" />
     116
     117The output produced by ''covmerge'' is actually a set of simple ASCII files that give a developer the necessary information to quickly determine the current status of the Code Coverage and enough information to determine the location of the uncovered code.  The following set of files is produced by ''covmerge''.
     118
     119||= File Name =||= Purpose of File =||
     120|| configuration.txt || Details the settings for the coverage run ||
     121|| summary.txt || Provides a summary of the results of the coverage run ||
     122|| sizes.txt || Provides a list identifying the file name and source line number of each uncovered range along with its size in bytes ||
     123|| report.txt || Details the settings for the coverage run ||
     124|| Explanations.txt.!NotFound || Contains the Explanations that were not found for this coverage run (see [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] for more information about how and why to use Explanations) ||
     125|| annotated.dmp || Provides the disassembled listing of hello.exe with indications of the object code that was not executed ||
     126|| hello.num || The symbol table of hello.exe ||
     127
     128You may wonder why the annotated disassembly (''annotated.dmp'') and symbol table (''hello.num'') are from hello.exe.  Because the set of object code to analyze is the same in all tests and linked to the same address range, the disassembly and symbol table for the analyzable portion of all executables is the same.
     129
     130= What was Discovered =
     131
     132
     133When we began the RTEMS Code Coverage effort, we performed coverage analysis on the development head of RTEMS 4.8 using the Baseline/-Os/POSIX Enabled Profile.  Some of our initial observations were interesting.  First, we were a little surprised at the incompleteness of the test suite.  We knew that there were some areas of the RTEMS code that were not tested at all, but we also found that areas we thought were tested were only partially tested.  We also observed some interesting things about the code we were analyzing.  We noticed that the use of inlining sometimes caused significant branch explosion.  This generated a lot of uncovered ranges that really mapped back to the same source code.  We also found that some defensive coding habits and coding style idioms could generate unreachable object code.  Also, the use of a case statement that includes all values of an enumerated type instead of an if statement sometimes lead to unreachable code.
     134
     135Other observations were related to the performance of the ''covmerge'' tool.  Of particular interest was the handling of NOP instructions.  Compilers can use NOP instructions to force alignment between functions or to fill delay-slots required by the processor.  Of course the NOP instructions are not executed and thus had a negative impact on the coverage.  The first attempt at dealing with NOP instructions was to mark them all as EXECUTED.  This was correct for the NOPs used for function alignment, but not for NOPs used for delay-slots.  Marking delay-slot NOPs as EXECUTED produced an unwanted side effect of occasionally spliting an uncovered range into two ranges.  We finally settled on an improved method for dealing with NOPs where NOPs were marked as EXECUTED unless they were between two NOT EXECUTED instructions.  An example is shown below:
     136
     137{{{
     1382003ee8:  80 a6 20 00   cmp  %i0, 0                            <== NOT EXECUTED
     1392003eec:  02 80 00 06   be  2003f04 <IMFS_fifo_write+0x60>     <== NOT EXECUTED
     1402003ef0:  01 00 00 00   nop                                    <== NOT EXECUTED
     1412003ef4:  40 00 78 fb   call  20222e0 <__errno>                <== NOT EXECUTED
     1422003ef8:  b0 20 00 18   neg  %i0                               <== NOT EXECUTED
     143}}}
     144
     145This solution to the NOP problem was important because NOPs were falsely increasing the number of uncovered ranges.  This created an unnecessary explosion of the reports and increased the uncovered ranges to examine.
     146= Resolving Uncovered Code =
     147
     148
     149The output files produced by ''covmerge'' are intended to provide both a quick-look at the status of a coverage run and the details needed to resolve the uncovered ranges.  As we worked through the resolution of the uncovered ranges, we noticed that the uncovered ranges usually fit into one of the following categories:
     150
     151 *  A new test case is needed.
     152
     153 *  Code unreachable in selected RTEMS configuration.  For example, the !SuperCore could have a feature only exercised by a POSIX API object.  It should be disabled when POSIX is not configured.
     154
     155 *  Debug or sanity checking code which should be placed inside an RTEMS_DEBUG conditional.
     156
     157 *  Unreachable paths generated for switch statements.  If the switch is based upon an enumerated type and the switch includes cases for all values, then it must be possible to actually generate all values at this point in the code. You can restructure the switch to only include possible values and thus avoid unreachable object code.  This is sometimes best done by rewriting the switch into a series of if/else statements.
     158
     159 *  Critical sections which are synchronizing actions with ISRs.  Most of these are very hard to hit and may require very specific support from a simulator environment.  OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner.  Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test.  The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.
     160
     161In general, it is interesting to note that the resolution of uncovered code does not simply translate into additions to the test suite.  Often the resolution points to improvements or changes to the analyzed code.  This can lead to more intelligent factoring of the code or a code re-design that produces a simpler solution.  There is also the notion that just because the analyzed code is "good" the way it is does not mean that it should not be rewritten to improve its testability.  Code that is completely tested is '''always''' better.
     162= Measuring Progress =
     163
     164
     165As mentioned above, the ''covmerge'' program produces reports that contain several metrics that can be used to measure progress.  The first is the number of uncovered object code ranges.  The second is the percent of untested object code as a percentage of the total object code size under analysis.  Together these metrics provide useful information about the status or progress of the Object Code Coverage.
     166
     167When we started the RTEMS Code Coverage effort, we did not immediately capture results to measure progress.  This actually ended up being the correct thing to do since the ''covmerge'' tool was in development and often produced results that were not directly comparable.  Now that the development of ''covmerge'' has largely settled, we can perform coverage runs on several RTEMS release points and see the progress of the coverage effort.  The results shown below were of the Baseline/-Os/POSIX Enabled Profile run on the SPARC/ERC32.
     168
     169||= Release =||= Covered % =||= Uncovered Ranges =||= Uncovered Bytes =||= Total Bytes =||
     170|| 4.7 || 77.51 || 454 || 17508 || 77840 ||
     171|| 4.8 || 76.37 || 538 || 21772 || 92140 ||
     172|| 4.9 || 96.41 || 167 || 2532 || 70564 ||
     173|| 4.10 (head 09/09/2009) || 100 || 0 || 0 || 70480 ||
     174
     175Several interesting facts can be seen from the data in the table.  There was no organized effort to perform coverage analysis prior to the 4.8 release.  This is evident in that there was no measurable improvement in coverage between 4.7 and 4.8.  The unassisted developer is just not going to recognize the need for more test cases in the test suite.  The coverage analysis began prior to the 4.9 release.  Not surprising, the progress was significant between 4.8 and 4.9.  At that time we addressed large uncovered ranges by doing simple things like adding test cases to the test suite and disabling code that was not used by the chosen configuration.  The last 3.5% of uncovered code was much harder to address, but the development head has now achieved 100% coverage.
     176
     177Now that we have achieved 100% Code Coverage using the Baseline/-Os/POSIX Enabled Profile, we would like to keep it 100% covered.  We have setup a periodic run of the coverage analysis against the development head.  The results are captured (http://rtems/ftp/pub/rtems/people/joel/coverage/) and can be monitored to ensure that future modifications to the analyzed code base do not produce uncovered code.
     178= Coverage Profiles =
     179
     180
     181RTEMS contains a lot of source code and although the primary focus of coverage analysis is to achieve 100% coverage of well-defined code subsets, we would also like to increase the amount of source code analyzed.  In order to manage the increase in a systematic manner, we defined two basic groups of source code.  The first group is called Baseline and the second group is called Developmental.  The Baseline group contains the source code that has achieved (or nearly achieved) 100% Object Code Coverage.  The Developmental group contains the source code for which there are very few test cases and therefore very poor coverage.
     182
     183Initially, the Baseline group included source code from the cpukit.  Specifically the following cpukit directories were included:  score, sapi, rtems and posix.  This group represents a full tasking and synchronization feature set.  What was not in the Baseline group was placed in the Developmental group.  The Developmental group included: libcsupport, libfs/imfs, libmisc/stackchk, libmisc/cpuuse, libmisc/bspcmdline, libmisc/dmpbuf and libmisc/devnull.
     184
     185Within the two groups, we recognized the need to use different compiler optimization levels and to analyze each group with POSIX threads enabled and POSIX threads disabled.  Applying these options produced eight sub-groups that we called profiles.  The eight profiles are:
     186
     187 *  Baseline/-Os/POSIX Enabled
     188 *  Baseline/-O2/POSIX Enabled
     189 *  Baseline/-Os/POSIX Disabled
     190 *  Baseline/-O2/POSIX Disabled
     191 *  Developmental/-Os/POSIX Enabled
     192 *  Developmental/-O2/POSIX Enabled
     193 *  Developmental/-Os/POSIX Disabled
     194 *  Developmental/-O2/POSIX Disabled
     195
     196Over time it is desirable to migrate code from the Developmental group to the Baseline.  As support libraries in cpukit become nearly 100% covered, they will be move from the Developmental group to the Baseline group.  Eventually, the Baseline group should contain all of the RTEMS code and the Developmental group should contain nothing.
     197= Compilation and Configuration Options =
     198
     199
     200The compilation level and POSIX configuration options are passed as command line arguments to the RTEMS Coverage Scripts.  [wiki:Developer/Coverage/HowTo RTEMS Code Coverage How To] provides details concerning how to run the RTEMS Coverage Scripts.  When we started the RTEMS Code Coverage effort, the code analyzed was compiled with optimization level -Os.  This optimizes for size without making the object code too difficult to follow.  Following the object code is important when trying to determime how to resolve the uncovered code.  Once the analyzed code approaches 100% covered, it is desirable to change the optimization level to -O2.  This is the most often used optimization level.
     201
     202Enabling or disabling POSIX allows us to analyze the RTEMS code in its two most commonly used threading configurations.  When POSIX is enabled, RTEMS is configured to use POSIX threads and the POSIX tests are built and executed as part of the test suite.  When POSIX is disabled, RTEMS is configured to use Classic RTEMS threads and the POSIX tests are not included in the test suite.
     203= Internal Compilation and Configuration Options =
     204
     205
     206There are several compilation and configuration options that are built into the RTEMS Coverage Scripts and are not selectable from the command line.  These options effect the RTEMS build and are used to simplify the code to aid analysis.  Ideally, we would like the coverage build to match the default build for RTEMS.  Over time, we will work to eliminate the need for the internal options.  The current options being used are:
     207
     208 *  NDEBUG=1 - Disables asserts.  We will probably keep this option.
     209 *  RTEMS_DO_NOT_INLINE_THREAD_ENABLE_DISPATCH=1 - Inlining resulted in branch explosion.  Over 200 new test cases will be needed to eliminate this option.
     210 *  RTEMS_DO_NOT_INLINE_CORE_MUTEX_SEIZE=1 - Inlining resulted in very difficult code to analyze.  This option should be able to be eliminated.
     211 *  RTEMS_DO_NOT_UNROLL_THREADQ_ENQUEUE_PRIORITY=1 - Unrolling loop resulted in multiple interrupt critical sections.  This option should be able to be eliminated.
     212= Beyond Object Code Coverage =
     213
     214
     215At this point, the RTEMS Code Coverage effort has been focused on Object Code Coverage.  But we would like to go beyond Object Code Coverage and address other traditional coverage criteria (see [wiki:Developer/Coverage/Theory Coverage Analysis Theory]).  We would also like to remain true to our original guidelines of using existing tools and performing the analysis without modifying the code to analyze.
     216= Achieving Statement Coverage =
     217
     218
     219Achieving Statement Coverage requires knowing which source files are involved (which ''covoar'' does) and which lines in those files can produce assembly code (which I don't think ''covoar'' can).  If any dead source code in RTEMS is detected by the combination of  gcc and Coverity Scan, then we can assume that all source code in RTEMS is represented in the generated executables. 
     220
     221The current object coverage utility ''covoar'' reports on which source lines were covered.  It could easily be modified to generate a report indicating which source lines were covered, not covered, or only partially covered.
     222
     223''covoar'' could also generate a bitmap per source file where the bit index indicates if a source line in that file was executed or not.  If we can generate a similar bit map from the source code which marks comments and other non-executable source lines as covered, then the union of the two bitmaps can be used to generate a report showing which source lines are not covered or represented in the object code.  This may indicate dead code or weaknesses in the tests.
     224
     225Adding a statement coverage report to ''covoar'' is an open project.
     226= Achieving !Condition/Decision Coverage =
     227
     228
     229Achieving !Condition/Decision Coverage requires knowing whether each branch has been both taken and not taken.  Currently QEMU and tsim can be used to gather this information.
     230
     231tsim produces bitmaps indicating instruction executed, branch taken, and branch not taken.
     232
     233All versions of QEMU produce a debug log of the instructions executed when an executable is run.  The trace information is analyzed to identify branch instructions and to determine whether the branch was taken and/or not taken.  Some versions of QEMU may also be able to produce a trace log which is denser but contains the same information.
     234
     235skyeye does not produce branch taken/not taken information.
     236
     237''covoar'' produces reports on which branch instructions are taken and not taken.  Our goal is to ensure that each branch instruction is taken and not taken. 
     238
     239GCC does not include debug information which indicates that a sequence of compare and branch instructions are part of a single logical condition.  This hinders our ability to augment ''covoar'' to make direct claims regarding Decision Coverage (DC) and Modified condition/decision coverage (MC/DC).
     240
     241We believe that for single condition ''if'' statements such as ''if (cond) action'' or ''if (cond) action1 else action2'', that we are achieving full DC and MC/DC coverage because all logical paths are exercised.
     242
     243Similarly given a dual OR condition ''if'' statement (in C) such as one the following:
     244
     245{{{
     246Case OR1: if (cond1 or cond2)
     247            action
     248Case OE2: if (cond1 or cond2)
     249            action1
     250          else
     251            action2
     252}}}
     253
     254We aim for the following cases given our branch coverage requirements:
     255
     256 *  cond1 branch taken, cond2 short-circuited
     257 *  cond1 branch not taken, cond2 taken
     258 *  cond1 branch not taken, cond2 not taken
     259
     260As the above set of test cases represent the entire set of possible execution paths, we have achieved DC and MC/DC level coverage.
     261
     262{{{
     263Case AND1: if (cond1 and cond2)
     264             action
     265Case AND2: if (cond1 and cond2)
     266             action1
     267           else
     268             action2
     269}}}
     270
     271We aim for the following cases given our branch coverage requirements:
     272
     273 *  cond1 branch taken, cond2 taken
     274 *  cond1 branch taken, cond2 not taken
     275 *  cond1 branch not taken, cond2 short-circuited
     276
     277Again, the above set of test cases represent the entire set of possible execution paths, we have achieved DC and MC/DC level coverage.
     278
     279Open projects in this area include:
     280
     281 *  proving our branch coverage testing policy meets decision coverage (DC) requirements in a more general sense.
     282 *  extending GCC to provide the debug information required to let covoar evaluate DC and MC/DC in C programs.
     283  *  IDEA: If GCC reliably reports that all conditions within a single ''if condition'' have the same line number, then we can use that information as the basis for the analysis.  Did we execute the proper set of cases for all branch instructions associated with a single debug line number.
     284= Current Status =
     285
     286
     287The [wiki:Developer/Coverage/Status Code Coverage Status] section lists the RTEMS BSPs on which we are performing (or would like to perform) Object Code Coverage.  We would like to continue to grow this list.  If you know of a simulator that includes coverage analysis, please let us know.
     288
     289With the instruction level coverage of core of RTEMS (e.g. score, rtems, posix, and sapi directories) near 100%, we have expanded our attention to include other non-networking portions of the cpukit.  The best way to find out which portions of the cpukit are not currently being included in coverage analysis is to look at the commented out lines calling ''filter_nm()'' in the method ''generate_symbols()'' in [http://www.rtems.org/viewvc/rtems-testing/rtems-coverage/do_coverage?view=markup rtems-testing/rtems-coverage/do_coverage]
     290
     291If you are interested in writing some simple parameter check error cases, then take a look at the branch taken/not taken coverage reports for the "core configuration". Some of these are a simple matter of adding missing test cases for bad parameter path.  Other cases are more difficult.  So if you run into trouble with the analysis, ask or skip it. A common pattern is this:
     292
     293{{{
     294if (arg1 bad)
     295  return EINVAL;
     296if (arg2 bad)
     297  return EINVAL;
     298}}}
     299
     300GCC is smart enough to optimize the returns into one block of code.  Thus we could have a test for arg1 or arg2 bad and obtain 100% instruction coverage.  But would not get 100% branch coverage.
     301
     302Initial analysis has been done at -Os which instructs gcc to generate smaller object code.  At -O2 which optimizes for speed, more code is generated and it is often clear looking at the -O2 reports, that there are test cases needed which are not required at -Os.
     303
     304== Coverage Analysis Theory ==
     305
     306The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intended to summarise or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
     307
     308The ultimate goal of Code Coverage Analysis is to ensure that a test suite adequately tests a particular body of code.  In order to achieve this goal, several different coverage criteria may have to be examined.  Let's consider the following criteria:
     309
     310 *  '''Statement Coverage''' - Has each line of the source code been executed?
     311 *  '''Decision Coverage''' (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
     312 *  '''Condition Coverage''' - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
     313 *  '''Object Coverage''' - Has each line of generated assembly been executed?
     314
     315=== Statement Coverage ===
     316
     317
     318Statement Coverage requires that each line of source code must be executed.  This is often considered the simplest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for identifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
     319
     320=== Decision Coverage ===
     321
     322
     323Decision Coverage requires that each control structure evaluate to both TRUE and FALSE.  This is a pretty good criteria because it generally ensures that both the TRUE and FALSE paths of an expression are covered.  However, short-circuit operators will prevent some portions of a complex expression from being evaluated.
     324=== Condition Coverage ===
     325
     326
     327
     328Condition Coverage requires that each boolean sub-expression evaluate to both TRUE and FALSE.  This criteria goes a little further than Decision Coverage by ensuring that the component parts of a compound expression each evaluate to TRUE and FALSE.  But it should be noted that Condition Coverage by itself does not necessarily imply decision coverage.  Because of this fact, it is best to apply Decision Coverage and Condition Coverage together.
     329
     330=== Object Coverage ===
     331
     332
     333Object Coverage requires that each line of generated assembly be executed.  This can be a very good general criteria because it ensures most of the test cases that the other criteria ensure.
     334
     335=== Criteria Relationships ===
     336
     337
     338[[Image(CoverageCategories.png)]]
     339
     340Each of these criteria can be used independently to analyze the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completeness and complexity of the test suite increases as it satisfies first Statement Coverage and then Decision Coverage and finally !Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and !Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy !Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and !Condition/Decision Coverage is the most complex to satisfy.
     341
     342=== An Example ===
     343
     344
     345In order to illustrate what is covered by each of the different criteria, consider the following example showing the source code for a simple if statement along with its generated pseudo-code instructions.
     346
     347|| '''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code''' ||
     348|| A || if (x OR y) || A1 || cmp x, 0 branch if FALSE to do something ||
     349|| || || A2 || cmp y, 0 branch if TRUE around do something ||
     350|| B || do something || B || do something instructions ||
     351
     352==== Statement Coverage ====
     353
     354
     355A single test case that allows the if statement to evaluate to TRUE will execute blocks A and B.  This will achieve 100% Statement Coverage.
     356
     357==== Decision Coverage ====
     358
     359
     360A minimum of two test cases are required to achieve 100% Decision Coverage.  One case must force the if statement to evaluate to TRUE and the other case must force the if statement to evaluate to FALSE.  A test case that forces a TRUE outcome will either execute blocks A1 and B or A1, A2 and B.  A test case that forces a FALSE outcome will execute blocks A1 and A2.
     361
     362==== !Condition/Decision Coverage ====
     363
     364
     365A minimum of two test cases are required to achieve 100% !Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
     366
     367==== Object Coverage ====
     368
     369
     370One carefully chosen test case where x is FALSE and y is TRUE will achieve 100% Object Coverage.  The test case will execute blocks A1, A2 and B.
     371
     372{{{#!comment
     373TBD : MC/DC Analysis : give a technical description on MC/DC .UNDER THEORY
     374}}}
     375
     376
     377
    423378= References =
    424379=== General Coverage Testing