Opened on 03/31/17 at 04:43:37

Closed on 06/05/20 at 04:32:03

#2962 closed defect (fixed)

Set test configurations to reflect test results.

Reported by: Chris Johns Owned by:
Priority: normal Milestone: 5.1
Component: test Version: 5
Severity: blocker Keywords: testsuite testing GCI
Cc: Blocked By:
Blocking:

Description

Tests can be set to the result we expect such as expected-fail, user-input and indeterminate as well as excluded so we can maintain accurate results for the testsuite.

This ticket covers setting of the correct state for all tests for tier 1 BSPs. Please add an "Updates #" for this ticket to any related commits and once done we can close this ticket.

Note: I have add a top level test configuration file called testsuites/rtems.tcfg that lets us specify a test configuration for all BSPs.

Change History (17)

comment:1 Changed on 10/10/17 at 06:46:55 by Sebastian Huber

Component: testing → unspecified

comment:2 Changed on 10/12/17 at 02:47:16 by Joel Sherrill

Owner: changed from joel.sherrill@… to Chris Johns
Status: new → assigned

comment:3 Changed on 11/09/17 at 06:27:14 by Sebastian Huber

Milestone: 4.12.0 → 5.1

Milestone renamed

comment:4 Changed on 10/14/18 at 01:10:08 by Joel Sherrill

Keywords: GCI added

Get Chris to write GCI instructions.

comment:5 Changed on 10/14/18 at 22:11:26 by Chris Johns

  1. Run the test suite for a bsp.
  2. Get the list of failures.
  3. Update or create a bsps/<arch>/<bsp>/config/<bsp>-testsuite.tcfg adding a line for each failure we expect to get with: 
    expected: <test-name>

comment:6 Changed on 10/19/18 at 00:26:06 by Chris Johns

Owner: Chris Johns deleted

comment:7 Changed on 11/15/18 at 22:58:12 by Chris Johns

Severity: normal → blocker

We need to document in the tests which tests are known to fail for specific archs or BSPs. It is important for us to be able to tell a user this is what we have baselined in a release.

comment:8 Changed on 11/22/18 at 02:09:42 by Chris Johns

The loopback test needs to be set to USER-INPUT to avoid a timeout when testing the samples with a network build.

comment:9 Changed on 04/28/20 at 05:33:14 by Chris Johns

Looking at the build reports I a differences with the erc32 BSP between no debug and RTEMS_DEBUG. Should the headwalk test like it does?

​https://lists.rtems.org/pipermail/build/2020-April/014203.html

Version 0, edited on 04/28/20 at 05:33:14 by Chris Johns (next)

comment:10 Changed on 04/29/20 at 04:58:33 by Sebastian Huber <sebastian.huber@…>

In a8f0d94/rtems:

libtests/heapwalk: Fix for RTEMS_DEBUG

Update #2962.

comment:11 Changed on 05/05/20 at 23:56:58 by Chris Johns

Component: unspecified → test

comment:12 Changed on 05/06/20 at 21:00:33 by Chris Johns <chrisj@…>

In 89f57a66/rtems:

testsuite: Add the BSP architecture to the include path

Updates #2962

comment:13 Changed on 05/06/20 at 21:00:37 by Chris Johns <chrisj@…>

In 7d00247/rtems:

testsuite: Add expected-fail to erc32, leon2, and leon3 BSPs

Updates #2962

comment:14 Changed on 05/06/20 at 21:00:44 by Chris Johns <chrisj@…>

In 1b1755d9/rtems:

testsuite: Add expected-fail to psim

Updates #2962

comment:15 Changed on 05/06/20 at 21:00:48 by Chris Johns <chrisj@…>

In 7e3af67/rtems:

testsuite: Add expected-fail to xilinx's zedboard, a9_qemu, zc702 and zc706

Updates #2962

comment:16 Changed on 05/06/20 at 21:00:51 by Chris Johns <chrisj@…>

In 084ea83/rtems:

testsuite: Add expected-fail to beagleboneblack

Updates #2962

comment:17 Changed on 06/05/20 at 04:32:03 by Sebastian Huber

Resolution: → fixed
Status: assigned → closed

The failures on powerpc/psim are due to #3982. For the psxfenv01 failures see #3996, #3997, and #3998. For arm/beagleboneblack see #3999. For the arm/xilinx-zynq see #4000. For dl06 on arm see #4001.

All test failures which need to be looked at have now tickets.

Note: See TracTickets for help on using tickets.