wiki:TBR/UserManual/RTEMS_Coverage_Analysis

Version 9 (modified by JoelSherrill, on Aug 18, 2009 at 4:02:40 AM) (diff)

/* Test Theory */ Add a description of theory.. statement coverage versus assembly coverage.

RTEMS Coverage Analysis

RTEMS is used in many critical systems. It is important that the RTEMS Project ensure that the RTEMS product is tested as thoroughly as possible. In this light, we want to ensure that as close to 100% of the generate assembly code is executed by the RTEMS test suite. We perform automated coverage testing using an processor simulator in conjunction with a set of RTEMS specific support scripts.

There are multiple ways to measure progress on this task. We primarily use two metrics. The first is the reduction in the number of uncovered binary code ranges from that identified initially. The second is the percent of untested binary object code as a percentage of the total code size under analysis. Together the metrics provide useful information. Some uncovered ranges may be a single instruction so eliminating that case improves the first metric more than the second.

Test Theory

The goal of the coverage analysis testing is to ensure that every line of generated assembly in a particular configuration is executed by the test suite. This is beyond statement level coverage in that we have to verify both paths of a C line like the following is executed:

x = (y) ? z : a;

That is one statement but includes a branch. That makes it one point to cover when viewed using statement coverage. But it is two points to cover in our full coverage of generated assembly.

Full Branch Coverage

TBD

Reasons Code is Not Executed

The coverage analysis provides a report on the ranges of assembly instructions within RTEMS subsystems which are not currently exercised by the tests in the current configuration. Each case has to be individually analysed and addressed. Historically, we have identified multiple categories for code being uncovered:

  • Needs a new test case
  • Unreachable in current RTEMS configuration. For example, the SuperCore? could have a feature only exercised by a POSIX API object. It could be disabled when POSIX is not configured.
  • Debug or sanity checking code which can be placed inside an RTEMS_DEBUG conditional.
  • Unreachable paths generated by gcc for switches. Sometimes you have to restructure switches to avoid unreachable object code.
  • Critical sections which are synchronizing actions with ISRs. Most of these are very hard to hit and may require very specific support from a simulator environment. OAR has used tsim to exercise these paths but this is not reproducible in a BSP independent manner. Worse, sometimes there is often no external way to know the case in question has been hit and no way to do it in a one shot test. The spintrcriticalXX and psxintrcriticalXX tests attempt to reproduce these cases.

Test Procedure

The scripts, tools, and patches are currently in the CVS module gcc-testing in the subdirectory rtems-coverage in the RTEMS CVS Repository.

Coverage Profiles

RTEMS includes a lot of source code and the coverage analysis should focus on improving the test coverage of well-defined code subsets.

Classic and POSIX

This is the first profile we tested. This focused on the score, sapi, rtems, and posix directories in the cpukit directory. This profile represents a full tasking and synchronization feature set.

Classic Only

In this profile, we disable POSIX and focus on the contents of the score, sapi, and rtems directories in the cpukit directory. The POSIX API and tests are disabled. In this profile, we expect to identify:

  • features in score only exercised by POSIX
  • features in score available via Classic API but only tested via POSIX
  • POSIX features like sleep() which are enabled with POSIX threads are disabled.

The first case will allow us to disable score features in this configuration and reduce the code size.

The second case allows us to approach 100% coverage in every RTEMS configuration.

Classic, POSIX, and Other CPUKit Components (experimental)

This is an experimental/developmental coverage configuration and adds almost all of the CPUKit contents that are non-networked. It nearly doubles the size of the code being covered. We are aiming for the entire contents of libcsupport, libmisc, and various filesystems. This is a large body of code and components like Termios and the file systems will require creativity to get automated coverage near 100%.

We have done initial tests on this profile. There is work to be done improving the test coverage. As components are covered 100%, they will be moved from experimental/developmental status to be included in the official coverage run.

We welcome your contributions.

BSPs Analyzed

If you know of a simulator that includes coverage analysis, please let us know.

ARM

The SkyEye project has added coverage analysis capabilities per our specifications. We are currently using it on some ARM targets to generate coverage reports.

EDB7312

GumStix?

Coldfire

Skyeye supports The MCF5206 but is currently unable to run the RTEMS BSP. Work to improve Skyeye's Coldfire support is welcomed.

=i386 =

pc386

We have identified using Qemu for the information. This project (http://libre.adacore.com/libre/tools/coverage/) aims to add the necessary capabilities to that simulator.

==SPARC==

We are using TSIM from Gaisler Research on all SPARC CPUs.

====ERC32====

====LEON2====

====LEON3====

Attachments (1)

Download all attachments as: .zip