Opened on 09/09/20 at 00:21:59

Last modified on 07/27/23 at 16:37:38

#4072 assigned defect

testsuite: Add ticket ids for expected-fail and indeterminate states, add annotations

Reported by: Chris Johns Owned by: Chris Johns
Priority: high Milestone: 6.1
Component: test Version: 6
Severity: blocker Keywords: test rtems-test
Cc: Blocked By:
Blocking:

Description

  • Require a ticket id be present for any expected-fail and indeterminate test states.
    1. Parse the ticket number from the .tcgf file and pass into the test.
    2. Have rtems-test obtain the ticket number from the test when run and provide the ticket number in the results.
  • Add annotations for tests. The list is to be provided by Joel.

Change History (6)

comment:1 Changed on 11/29/22 at 22:18:54 by Chris Johns

Priority: normal → high
Severity: normal → blocker

comment:2 Changed on 06/29/23 at 23:34:15 by Chris Johns

This is now updated in the spec YAML files.

comment:3 Changed on 06/29/23 at 23:40:04 by Kinsey Moore

I have some to handle for AArch64 and ARM and any BSPs that run specifically on QEMU.

comment:4 Changed on 07/10/23 at 13:49:21 by Kinsey Moore <kinsey.moore@…>

In f46c15f/rtems:

spec: Add QEMU test annotations

QEMU is known to fail certain tests intermittently due to clock tick
delivery issues. This defines those tests as intermittent for BSPs
intended to run on QEMU alone.

Updates #4922
Updates #4072

comment:5 Changed on 07/10/23 at 13:52:26 by Kinsey Moore

The above commit takes care of all the indeterminate test states that I'm aware of in the AArch64 realm.

comment:6 Changed on 07/27/23 at 16:37:38 by Kinsey Moore

The only indeterminate test state is now documented properly. These are the remaining files that have a test state of expected-fail:

spec/build/bsps/tst-xfail-intr.yml (validation-intr for powerpc/psim)
spec/build/bsps/i386/pc386/tst-xfail-val.yml (various validation tests for i386/pc386)
spec/build/bsps/tst-xfail-dl06.yml (dl06 for arm/imx7)
spec/build/bsps/tst.yml (psxfenv01 for all arches)

Note: See TracTickets for help on using tickets.