Changes between Version 17 and Version 18 of GCI/Documentation/CoverageAnalysis/Coverage


Ignore:
Timestamp:
Dec 8, 2018, 8:39:06 PM (6 months ago)
Author:
shashvat jain
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • GCI/Documentation/CoverageAnalysis/Coverage

    v17 v18  
    224224
    225225Adding a statement coverage report to ''covoar'' is an open project.
     226
     227= Coverage Analysis Theory =
     228
     229The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intended to summarise or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
     230
     231The ultimate goal of Code Coverage Analysis is to ensure that a test suite adequately tests a particular body of code.  In order to achieve this goal, several different coverage criteria may have to be examined.  Let's consider the following criteria:
     232
     233 *  '''Statement Coverage''' - Has each line of the source code been executed?
     234 *  '''Decision Coverage''' (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
     235 *  '''Condition Coverage''' - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
     236 *  '''Object Coverage''' - Has each line of generated assembly been executed?
     237
     238=== Statement Coverage ===
     239
     240
     241Statement Coverage requires that each line of source code must be executed.  This is often considered the simplest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for identifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
     242
     243=== Decision Coverage ===
     244
     245
     246Decision Coverage requires that each control structure evaluate to both TRUE and FALSE.  This is a pretty good criteria because it generally ensures that both the TRUE and FALSE paths of an expression are covered.  However, short-circuit operators will prevent some portions of a complex expression from being evaluated.
     247=== Condition Coverage ===
     248
     249
     250
     251Condition Coverage requires that each boolean sub-expression evaluate to both TRUE and FALSE.  This criteria goes a little further than Decision Coverage by ensuring that the component parts of a compound expression each evaluate to TRUE and FALSE.  But it should be noted that Condition Coverage by itself does not necessarily imply decision coverage.  Because of this fact, it is best to apply Decision Coverage and Condition Coverage together.
     252
     253=== Object Coverage ===
     254
     255
     256Object Coverage requires that each line of generated assembly be executed.  This can be a very good general criteria because it ensures most of the test cases that the other criteria ensure.
     257
     258=== Criteria Relationships ===
     259
     260
     261[[Image(CoverageCategories.png)]]
     262
     263Each of these criteria can be used independently to analyze the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completeness and complexity of the test suite increases as it satisfies first Statement Coverage and then Decision Coverage and finally !Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and !Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy !Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and !Condition/Decision Coverage is the most complex to satisfy.
     264
     265=== An Example ===
     266
     267
     268In order to illustrate what is covered by each of the different criteria, consider the following example showing the source code for a simple if statement along with its generated pseudo-code instructions.
     269
     270|| '''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code''' ||
     271|| A || if (x OR y) || A1 || cmp x, 0 branch if FALSE to do something ||
     272|| || || A2 || cmp y, 0 branch if TRUE around do something ||
     273|| B || do something || B || do something instructions ||
     274
     275==== Statement Coverage ====
     276
     277
     278A single test case that allows the if statement to evaluate to TRUE will execute blocks A and B.  This will achieve 100% Statement Coverage.
     279
     280==== Decision Coverage ====
     281
     282
     283A minimum of two test cases are required to achieve 100% Decision Coverage.  One case must force the if statement to evaluate to TRUE and the other case must force the if statement to evaluate to FALSE.  A test case that forces a TRUE outcome will either execute blocks A1 and B or A1, A2 and B.  A test case that forces a FALSE outcome will execute blocks A1 and A2.
     284
     285==== !Condition/Decision Coverage ====
     286
     287
     288A minimum of two test cases are required to achieve 100% !Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
     289
     290==== Object Coverage ====
     291
     292
     293One carefully chosen test case where x is FALSE and y is TRUE will achieve 100% Object Coverage.  The test case will execute blocks A1, A2 and B.
     294
     295{{{#!comment
     296TBD : MC/DC Analysis : give a technical description on MC/DC .UNDER THEORY
     297}}}
     298
    226299= Achieving !Condition/Decision Coverage =
    227300
     
    302375Initial analysis has been done at -Os which instructs gcc to generate smaller object code.  At -O2 which optimizes for speed, more code is generated and it is often clear looking at the -O2 reports, that there are test cases needed which are not required at -Os.
    303376
    304 == Coverage Analysis Theory ==
    305 
    306 The subject of Code Coverage Analysis is broad and has been written about many times over.  This background material is not intended to summarise or rehash what can be read elsewhere.  Instead, the focus here will be on the aspects of Code Coverage Analysis as they pertain to the [wiki:TBR/UserManual/RTEMS_Coverage_Analysis RTEMS Coverage Analysis] effort.
    307 
    308 The ultimate goal of Code Coverage Analysis is to ensure that a test suite adequately tests a particular body of code.  In order to achieve this goal, several different coverage criteria may have to be examined.  Let's consider the following criteria:
    309 
    310  *  '''Statement Coverage''' - Has each line of the source code been executed?
    311  *  '''Decision Coverage''' (also known as Branch coverage) - Has each control structure (such as an if statement) evaluated both to true and false?
    312  *  '''Condition Coverage''' - Has each boolean sub-expression evaluated both to true and false (this does not necessarily imply decision coverage)?
    313  *  '''Object Coverage''' - Has each line of generated assembly been executed?
    314 
    315 === Statement Coverage ===
    316 
    317 
    318 Statement Coverage requires that each line of source code must be executed.  This is often considered the simplest criteria.  The problem is that it only identifies the lines that were executed, and does not consider the logic flow of the code.  It can be useful for identifying "chunks" of code (i.e. new functionality) that are not covered by the test suite, but not much else.
    319 
    320 === Decision Coverage ===
    321 
    322 
    323 Decision Coverage requires that each control structure evaluate to both TRUE and FALSE.  This is a pretty good criteria because it generally ensures that both the TRUE and FALSE paths of an expression are covered.  However, short-circuit operators will prevent some portions of a complex expression from being evaluated.
    324 === Condition Coverage ===
    325 
    326 
    327 
    328 Condition Coverage requires that each boolean sub-expression evaluate to both TRUE and FALSE.  This criteria goes a little further than Decision Coverage by ensuring that the component parts of a compound expression each evaluate to TRUE and FALSE.  But it should be noted that Condition Coverage by itself does not necessarily imply decision coverage.  Because of this fact, it is best to apply Decision Coverage and Condition Coverage together.
    329 
    330 === Object Coverage ===
    331 
    332 
    333 Object Coverage requires that each line of generated assembly be executed.  This can be a very good general criteria because it ensures most of the test cases that the other criteria ensure.
    334 
    335 === Criteria Relationships ===
    336 
    337 
    338 [[Image(CoverageCategories.png)]]
    339 
    340 Each of these criteria can be used independently to analyze the code in question.  Application of any one criteria will likely improve the test suite to some degree albeit at the cost of increasing the complexity of the test suite.  Examination of the criteria collectively, shows that there are clear relationships between the different criteria as shown in the picture.  The completeness and complexity of the test suite increases as it satisfies first Statement Coverage and then Decision Coverage and finally !Condition/Decision Coverage.  If the test suite satisfies Statement Coverage, it will partially satisfy Decision Coverage and !Condition/Decision Coverage.  If the test suite satisfies Decision Coverage, it will completely satisfy Statement Coverage and partially satisfy !Condition/Decision Coverage.  Note the fact that Object Coverage satisfies part of all of the other criteria.  There is also a complexity relationship where Statement Coverage is the least complex to satisfy and !Condition/Decision Coverage is the most complex to satisfy.
    341 
    342 === An Example ===
    343 
    344 
    345 In order to illustrate what is covered by each of the different criteria, consider the following example showing the source code for a simple if statement along with its generated pseudo-code instructions.
    346 
    347 || '''Block''' || '''Source Code''' || '''Block''' || '''Object Pseudo-code''' ||
    348 || A || if (x OR y) || A1 || cmp x, 0 branch if FALSE to do something ||
    349 || || || A2 || cmp y, 0 branch if TRUE around do something ||
    350 || B || do something || B || do something instructions ||
    351 
    352 ==== Statement Coverage ====
    353 
    354 
    355 A single test case that allows the if statement to evaluate to TRUE will execute blocks A and B.  This will achieve 100% Statement Coverage.
    356 
    357 ==== Decision Coverage ====
    358 
    359 
    360 A minimum of two test cases are required to achieve 100% Decision Coverage.  One case must force the if statement to evaluate to TRUE and the other case must force the if statement to evaluate to FALSE.  A test case that forces a TRUE outcome will either execute blocks A1 and B or A1, A2 and B.  A test case that forces a FALSE outcome will execute blocks A1 and A2.
    361 
    362 ==== !Condition/Decision Coverage ====
    363 
    364 
    365 A minimum of two test cases are required to achieve 100% !Condition/Decision Coverage.  In the first case, x and y must be TRUE.  In the second case, x and y must be FALSE.  The test case that forces a TRUE outcome will execute blocks A1 and B.  The test case that forces a FALSE outcome will execute blocks A1 and A2.
    366 
    367 ==== Object Coverage ====
    368 
    369 
    370 One carefully chosen test case where x is FALSE and y is TRUE will achieve 100% Object Coverage.  The test case will execute blocks A1, A2 and B.
    371 
    372 {{{#!comment
    373 TBD : MC/DC Analysis : give a technical description on MC/DC .UNDER THEORY
    374 }}}
    375 
    376 
    377377
    378378= References =
     379
    379380=== General Coverage Testing
    380381 *  [http://en.wikipedia.org/wiki/Code_coverage Code Coverage Definition]