Changes between Version 21 and Version 22 of Developer/Coverage/Theory


Ignore:
Timestamp:
Nov 22, 2014, 5:04:27 AM (5 years ago)
Author:
Chris Johns
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Developer/Coverage/Theory

    v21 v22  
    66
    77
    8 The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intented to summerize or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
     8The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intended to summarise or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
    99
    1010The ultimate goal of Code Coverage Analysis is to ensure that a test suite adequately tests a particular body of code.  In order to achieve this goal, several different coverage criteria may have to be examined.  Let's consider the following criteria:
     
    1414 *  '''Condition Coverage''' - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
    1515 *  '''Object Coverage''' - Has each line of generated assembly been executed?
     16
    1617= Statement Coverage =
    1718
    1819
    19 Statement Coverage requires that each line of source code must be executed.  This is often considered the simpliest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for indentifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
     20Statement Coverage requires that each line of source code must be executed.  This is often considered the simplest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for identifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
     21
    2022= Decision Coverage =
    2123
     
    2527
    2628
     29
    2730Condition Coverage requires that each boolean sub-expression evaluate to both TRUE and FALSE.  This criteria goes a little further than Decision Coverage by ensuring that the component parts of a compound expression each evaluate to TRUE and FALSE.  But it should be noted that Condition Coverage by itself does not necessarily imply decision coverage.  Because of this fact, it is best to apply Decision Coverage and Condition Coverage together.
     31
    2832= Object Coverage =
    2933
    3034
    3135Object Coverage requires that each line of generated assembly be executed.  This can be a very good general criteria because it ensures most of the test cases that the other criteria ensure.
     36
    3237= Criteria Relationships =
    3338
    3439
    35 [[Image(CoverageCategories.png)]]]]
     40[[Image(CoverageCategories.png)]]
    3641
    37 Each of these criteria can be used independently to analyze the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completness and complexity of the test suite increases as it satifies first Statement Coverage and then Decision Coverage and finally Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and Condition/Decision Coverage is the most complex to satisfy.
     42Each of these criteria can be used independently to analyse the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completeness and complexity of the test suite increases as it satisfies first Statement Coverage and then Decision Coverage and finally Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and Condition/Decision Coverage is the most complex to satisfy.
    3843
    39 <br style="clear: both" />
    4044= An Example =
    4145
     
    4347In order to illustrate what is covered by each of the different criteria, consider the following example showing the source code for a simple if statement along with its generated pseudo-code instructions.
    4448
    45 {| border="1" style="margin: 1em auto 1em auto;text-align: left;"
    46 |+
    47 |-
    48 |'''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code'''
    49 |-
    50 | A || if (x OR y) || A1 || cmp x, 0
    51 branch if FALSE to do something
    52 |-
    53 | || || A2 || cmp y, 0
    54 branch if TRUE around do something
    55 |-
    56 | B || do something || B|| do something instructions
    57 |-
    58 |}
     49|| '''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code''' ||
     50|| A || if (x OR y) || A1 || cmp x, 0 branch if FALSE to do something ||
     51|| || || A2 || cmp y, 0 branch if TRUE around do something ||
     52|| B || do something || B || do something instructions ||
     53
    5954= Statement Coverage =
    6055
    6156
    6257A single test case that allows the if statement to evaluate to TRUE will execute blocks A and B.  This will achieve 100% Statement Coverage.
     58
    6359= Decision Coverage =
    6460
    6561
    6662A minimum of two test cases are required to achieve 100% Decision Coverage.  One case must force the if statement to evaluate to TRUE and the other case must force the if statement to evaluate to FALSE.  A test case that forces a TRUE outcome will either execute blocks A1 and B or A1, A2 and B.  A test case that forces a FALSE outcome will execute blocks A1 and A2.
    67 = Condition/Decision Coverage =
     63
     64= !Condition/Decision Coverage =
    6865
    6966
    70 A minimum of two test cases are required to achieve 100% Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
     67A minimum of two test cases are required to achieve 100% !Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
     68
    7169= Object Coverage =
    7270