#2962 assigned defect

Set test configurations to reflect test results.

Reported by: Chris Johns Owned by:
Priority: normal Milestone: 5.1
Component: test Version: 5
Severity: blocker Keywords: testsuite testing GCI
Cc: Blocked By:
Blocking:

Description

Tests can be set to the result we expect such as expected-fail, user-input and indeterminate as well as excluded so we can maintain accurate results for the testsuite.

This ticket covers setting of the correct state for all tests for tier 1 BSPs. Please add an "Updates #" for this ticket to any related commits and once done we can close this ticket.

Note: I have add a top level test configuration file called testsuites/rtems.tcfg that lets us specify a test configuration for all BSPs.

Change History (16)

comment:1 Changed on Oct 10, 2017 at 6:46:55 AM by Sebastian Huber

Component: testingunspecified

comment:2 Changed on Oct 12, 2017 at 2:47:16 AM by Joel Sherrill

Owner: changed from joel.sherrill@… to Chris Johns
Status: newassigned

comment:3 Changed on Nov 9, 2017 at 6:27:14 AM by Sebastian Huber

Milestone: 4.12.05.1

Milestone renamed

comment:4 Changed on Oct 14, 2018 at 1:10:08 AM by Joel Sherrill

Keywords: GCI added

Get Chris to write GCI instructions.

comment:5 Changed on Oct 14, 2018 at 10:11:26 PM by Chris Johns

  1. Run the test suite for a bsp.
  2. Get the list of failures.
  3. Update or create a bsps/<arch>/<bsp>/config/<bsp>-testsuite.tcfg adding a line for each failure we expect to get with:
    expected: <test-name>
    

comment:6 Changed on Oct 19, 2018 at 12:26:06 AM by Chris Johns

Owner: Chris Johns deleted

comment:7 Changed on Nov 15, 2018 at 10:58:12 PM by Chris Johns

Severity: normalblocker

We need to document in the tests which tests are known to fail for specific archs or BSPs. It is important for us to be able to tell a user this is what we have baselined in a release.

comment:8 Changed on Nov 22, 2018 at 2:09:42 AM by Chris Johns

The loopback test needs to be set to USER-INPUT to avoid a timeout when testing the samples with a network build.

comment:9 Changed on Apr 28, 2020 at 5:33:14 AM by Chris Johns

Looking at the build reports I see a differences with the erc32 BSP between no debug and RTEMS_DEBUG. Should the headwalk test like it does?

https://lists.rtems.org/pipermail/build/2020-April/014203.html

Last edited on Apr 28, 2020 at 5:33:52 AM by Chris Johns (previous) (diff)

comment:10 Changed on Apr 29, 2020 at 4:58:33 AM by Sebastian Huber <sebastian.huber@…>

In a8f0d94/rtems:

libtests/heapwalk: Fix for RTEMS_DEBUG

Update #2962.

comment:11 Changed on May 5, 2020 at 11:56:58 PM by Chris Johns

Component: unspecifiedtest

comment:12 Changed on May 6, 2020 at 9:00:33 PM by Chris Johns <chrisj@…>

In 89f57a66/rtems:

testsuite: Add the BSP architecture to the include path

Updates #2962

comment:13 Changed on May 6, 2020 at 9:00:37 PM by Chris Johns <chrisj@…>

In 7d00247/rtems:

testsuite: Add expected-fail to erc32, leon2, and leon3 BSPs

Updates #2962

comment:14 Changed on May 6, 2020 at 9:00:44 PM by Chris Johns <chrisj@…>

In 1b1755d9/rtems:

testsuite: Add expected-fail to psim

Updates #2962

comment:15 Changed on May 6, 2020 at 9:00:48 PM by Chris Johns <chrisj@…>

In 7e3af67/rtems:

testsuite: Add expected-fail to xilinx's zedboard, a9_qemu, zc702 and zc706

Updates #2962

comment:16 Changed on May 6, 2020 at 9:00:51 PM by Chris Johns <chrisj@…>

In 084ea83/rtems:

testsuite: Add expected-fail to beagleboneblack

Updates #2962

Note: See TracTickets for help on using tickets.