#3199 accepted enhancement

New test framework

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: normal Milestone: 6.1
Component: lib Version: 5
Severity: normal Keywords: qualification
Cc: Blocked By:
Blocking: #3716

Description

Requirements:

  • Support for RTEMS and general POSIX systems such as Linux or FreeBSD
  • Test output functions
    • printf() like
    • No floating point support by default
    • Optional floating point support
    • Non-blocking
    • Arbitrary context, e.g. during system start, before device drivers, in interrupt context
  • Automatic test case registrations (constructors, linker set)
  • Standard test case output, e.g. begin/end of test case message
  • Ability to sort test cases at run-time
  • Safe assert functions, e.g. assert(a == b) vs. assert(a = b) vs. assert_eq(a, b)
  • Easy to write, e.g. avoid long namespace prefix rtems_test_*
  • Support multiple threads
  • No dependency on C++
  • Context validation, context invariant validation (e.g. thread dispatch disable level, interrupt state, heap state)
  • Test pattern support (e.g. spintercritical*, parallel tests on SMP)

No ready to use framework exists which fulfils these requirements. We have to write something for RTEMS.

API proposal (header file <T.h>):

T_eq(a, b)
T_gt(a, b)
T_printf(fmt, ...)
T_output_printf(putchar, fmt, ...)
T_output_string(putchar, s, n)
T_output_add_format_handler(handler, arg)

Example for T_output_add_format():

#include <stdio.h>
#include <string.h>

typedef void (*T_putchar)(char c);

typedef size_t (*T_format_handler)(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out);

int T_printf(const char *fmt, ...);

void T_output_printf(T_putchar putchar, const char *fmt, ...);

void T_output_string(T_putchar putchar, const char *s, size_t n);

void T_output_add_format_handler(T_format_handler handler, void *arg);

size_t T_output_float_handler(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out)
{
  char buf[32];

  if (strncmp(fmt, "%f", 2) != 0) {
    return 0;
  }

  *out = snprintf(buf, sizeof(buf), "%f", s);

  if (*out > 0) {
    T_output_string(putchar, s, (size_t) *out);
  }

  return 2;
}

void f(void)
{
  T_output_add_format_handler(T_output_float_handler, NULL);
}

Attachments (2)

t.tar.gz (26.5 KB) - added by Sebastian Huber on Nov 23, 2018 at 10:26:10 AM.
The T Test Framework
contributor.pdf (188.8 KB) - added by Sebastian Huber on Dec 6, 2018 at 2:47:59 PM.
Draft v4

Download all attachments as: .zip

Change History (75)

comment:2 Changed on Oct 28, 2017 at 12:08:04 PM by Sebastian Huber <sebastian.huber@…>

In f703e7f/rtems:

tests: Move rtems_test_printer definition

Statically initialize it to use printk().

Update #3170.
Update #3199.

comment:3 Changed on Oct 28, 2017 at 12:08:16 PM by Sebastian Huber <sebastian.huber@…>

In 7bec7f27/rtems:

rtems: Add rtems_print_printer_fprintf_putc()

Update #3170.
Update #3199.

comment:4 Changed on Oct 28, 2017 at 12:08:27 PM by Sebastian Huber <sebastian.huber@…>

In 73d892d8/rtems:

tests: Use rtems_test_printer

Update #3170.
Update #3199.

comment:5 Changed on Oct 28, 2017 at 12:08:39 PM by Sebastian Huber <sebastian.huber@…>

In 46ddc3c5/rtems:

tests: Use rtems_print_printer_fprintf_putc()

Use rtems_print_printer_fprintf_putc() instead of
rtems_print_printer_printf() to output via rtems_putc().

Update #3170.
Update #3199.

comment:6 Changed on Oct 28, 2017 at 12:08:51 PM by Sebastian Huber <sebastian.huber@…>

In 7e10291/rtems:

tests: Use rtems_test_printer in general

Update #3170.
Update #3199.

comment:7 Changed on Oct 28, 2017 at 12:09:02 PM by Sebastian Huber <sebastian.huber@…>

In acc9d064/rtems:

tests: Remove obsolete TESTS_USE_PRINTK

Update #3170.
Update #3199.

comment:8 Changed on Oct 28, 2017 at 12:09:15 PM by Sebastian Huber <sebastian.huber@…>

In af43554/rtems:

tests: Remove TEST_INIT

The TEST_EXTERN is a used only by the system.h style tests and they use
CONFIGURE_INIT appropriately.

Update #3170.
Update #3199.

comment:9 Changed on Oct 30, 2017 at 5:31:17 AM by Chris Johns <chrisj@…>

In 2126438a/rtems:

testsuite: Add bspIo for a local printk.

Update #3170.
Update #3199.

comment:10 Changed on Nov 2, 2017 at 1:25:12 PM by Sebastian Huber <sebastian.huber@…>

In 0d796d6/rtems:

tests: Delete obsolete TESTS_USE_PRINTF

Update #3170.
Update #3199.

comment:11 Changed on Nov 2, 2017 at 1:26:51 PM by Sebastian Huber <sebastian.huber@…>

In 8c1f4064/rtems:

tests: Use printf() instead of fprintf()

Update #3170.
Update #3199.

comment:12 Changed on Nov 6, 2017 at 6:29:13 AM by Sebastian Huber <sebastian.huber@…>

In 1082798/rtems:

score: Add _IO_Printf() and _IO_Vprintf()

The previous vprintk() implementation had a questionable licence header,
lacks support for the 'z' and 'j' format specifiers, is not robust
against invalid format specifiers, uses a global variable for output.
Replace it with a stripped down version of the FreeBSD kernel kvprintf()
function.

The new implementation allows a low overhead rtems_snprintf() if
necessary.

Update #3199.
Close #3216.

comment:13 Changed on Nov 6, 2017 at 6:29:25 AM by Sebastian Huber <sebastian.huber@…>

In ac28f15/rtems:

Add simple console driver

Update #3170.
Update #3199.

comment:14 Changed on Nov 6, 2017 at 6:29:41 AM by Sebastian Huber <sebastian.huber@…>

In c4b8b147/rtems:

tests: Use simple console driver

Update #3170.
Update #3199.

comment:15 Changed on Nov 7, 2017 at 6:09:15 AM by Sebastian Huber <sebastian.huber@…>

In 7b00c2fa/rtems:

tests: Use <tmacros.h> in all tests

Update #3170.
Update #3199.

comment:16 Changed on Nov 7, 2017 at 7:32:57 AM by Sebastian Huber <sebastian.huber@…>

In 32ceb38/rtems:

tests: Use <tmacros.h>

Update #3170.
Update #3199.

comment:17 Changed on Nov 10, 2017 at 2:42:58 AM by Chris Johns <chrisj@…>

In 6f3fb8a/rtems:

cpukit: Add a Version API.

Provide functions to get the version string, major, minor and revision
numbers and the version control identifer that is a unique tag for
the version control system.

Update #3199.

comment:18 Changed on Nov 11, 2017 at 5:24:06 AM by Chris Johns <chrisj@…>

In e6df806/rtems:

tests: Use ld to map (wrap) printf, puts and putchar to tester functions.

  • Remove the macro defines and the need for tmacro.h by remapping the symbols using ld's wrap option.
  • Remove FLUSH_OUTPUT, it was empty.
  • Move rtems_test_exit to libmisc/testsupport as a function.

Update #3199.

comment:19 Changed on Nov 15, 2017 at 7:10:55 AM by Sebastian Huber <sebastian.huber@…>

In 62119d2/rtems:

dl01, dl02, dl05: Fix unresolved printf symbol

The link time wrap of printf leads to unresolved symbols in the loadable
modules. This resulted in infinite loops and test timeouts. Use
rtems_printf() for output.

Update #3199.

comment:20 Changed on Nov 17, 2017 at 5:55:31 AM by Sebastian Huber <sebastian.huber@…>

In 727cf48/rtems:

sptests/spversion01: Simplify configuration

Update #3199.

comment:21 Changed on Feb 5, 2018 at 9:48:37 AM by Sebastian Huber <sebastian.huber@…>

In d078405/rtems-docs:

CONFIGURE_APPLICATION_NEEDS_SIMPLE_CONSOLE_DRIVER

Close #3170.
Update #3199.

Changed on Nov 23, 2018 at 10:26:10 AM by Sebastian Huber

Attachment: t.tar.gz added

The T Test Framework

Changed on Dec 6, 2018 at 2:47:59 PM by Sebastian Huber

Attachment: contributor.pdf added

Draft v4

comment:22 Changed on Mar 4, 2019 at 1:03:31 PM by Sebastian Huber

Blocking: 3716 added
Status: assignedaccepted

comment:23 Changed on Mar 27, 2019 at 6:29:28 AM by Sebastian Huber <sebastian.huber@…>

In 6fe01e4b/rtems:

build: Move test support to librtemstest.a

One reason to move the test support into a dedicated library are the
standard output wrap_*() functions. They may conflict with
application level wrappers.

Update #3199.

comment:24 Changed on Mar 27, 2019 at 6:29:32 AM by Sebastian Huber <sebastian.huber@…>

In cfcc2cbf/rtems:

Add RTEMS Test Framework

Update #3199.

comment:25 Changed on Mar 27, 2019 at 6:29:37 AM by Sebastian Huber <sebastian.huber@…>

In cbfc3415/rtems:

ttest01: New test

This is an example test using the RTEMS Test Framework. It tests also
the framework itself.

Add T_FILE_NAME command line define to get rid of the full file path.
This is important to reduce the read-only data of test files and make
them build system independent.

Update #3199.

comment:26 Changed on Mar 27, 2019 at 6:34:23 AM by Sebastian Huber <sebastian.huber@…>

In c2e582d/rtems-docs:

eng: Add software test framework chapter

Update #3199.

comment:27 Changed on Oct 11, 2019 at 9:02:08 AM by Sebastian Huber <sebastian.huber@…>

In f88025a/rtems:

ttest01: Adjust SPDX-License-Identifier

Update #3199.

comment:28 Changed on Oct 11, 2019 at 9:02:13 AM by Sebastian Huber <sebastian.huber@…>

In b406d071/rtems:

libtest: Do all output in test runner

This ensures that lines are output atomically if they are produced by
different other contexts, e.g. interrupts, other processors, other
threads.

Update #3199.

comment:29 Changed on Oct 11, 2019 at 9:02:18 AM by Sebastian Huber <sebastian.huber@…>

In feb27f9/rtems:

ttest01: Add more test cases

Update #3199.

comment:30 Changed on Oct 11, 2019 at 9:02:26 AM by Sebastian Huber <sebastian.huber@…>

In b5e61f95/rtems:

libtest: Add more action events

This allows more control over the initialization and finalization run.

Update #3199.

comment:31 Changed on Oct 11, 2019 at 9:02:32 AM by Sebastian Huber <sebastian.huber@…>

In e71f0a5/rtems:

ttest01: Check init/final run output

Update #3199.

comment:32 Changed on Feb 10, 2020 at 8:03:35 AM by Sebastian Huber <sebastian.huber@…>

In 45a0f65/rtems-docs:

eng: Document test framework formatted output

Update #3199.

comment:33 Changed on Feb 10, 2020 at 8:03:37 AM by Sebastian Huber <sebastian.huber@…>

In b6c61e3/rtems-docs:

eng: Mention test framework buffer configuration

Update #3199.

comment:34 Changed on Feb 10, 2020 at 12:49:18 PM by Sebastian Huber <sebastian.huber@…>

In 76b3aea/rtems-docs:

eng: Grammar fix

Update #3199.

comment:35 Changed on Jul 17, 2020 at 9:23:51 AM by Sebastian Huber

Milestone: Indefinite6.1

comment:36 Changed on Jul 23, 2020 at 8:06:12 AM by Sebastian Huber <sebastian.huber@…>

In 21fa28c/rtems-docs:

eng: Update test framework chapter

Document the dynamic text fixtures, utility functions, and the interrupt
test support. Reorder some sections and reword some paragraphs based on
review comments.

Update #3199.

comment:37 Changed on Jul 23, 2020 at 8:57:50 AM by Sebastian Huber <sebastian.huber@…>

In 9de8d61/rtems:

libtest: <rtems/test.h> to <rtems/test-info.h>

Rename this header file to later move <t.h> to <rtems/test.h>. The main
feature provided by <rtems/test-info.h> is the output of standard test
information which is consumed by the RTEMS Tester.

Update #3199.

comment:38 Changed on Jul 23, 2020 at 8:57:54 AM by Sebastian Huber <sebastian.huber@…>

In 361404e8/rtems:

libtest: Move <t.h> to <rtems/test.h>

Update #3199.

comment:39 Changed on Jul 23, 2020 at 8:57:57 AM by Sebastian Huber <sebastian.huber@…>

In e3e3b871/rtems:

libtest: Add T_busy()

Update #3199.

comment:40 Changed on Jul 23, 2020 at 8:58:00 AM by Sebastian Huber <sebastian.huber@…>

In af92665/rtems:

libtest: Add T_get_one_clock_tick_busy()

Update #3199.

comment:41 Changed on Jul 23, 2020 at 8:58:04 AM by Sebastian Huber <sebastian.huber@…>

In c081c68/rtems:

libtest: Add T_make_runner()

Update #3199.

comment:42 Changed on Jul 23, 2020 at 8:58:07 AM by Sebastian Huber <sebastian.huber@…>

In 6b27e32/rtems:

libtest: Support custom scope messages via fixture

Update #3199.

comment:43 Changed on Jul 23, 2020 at 8:58:10 AM by Sebastian Huber <sebastian.huber@…>

In cb3c6bdc/rtems:

libtest: Add push/pop fixture support

Update #3199.

comment:44 Changed on Jul 23, 2020 at 8:58:14 AM by Sebastian Huber <sebastian.huber@…>

In 63e4278/rtems:

libtest: Add T_get_scope()

Update #3199.

comment:45 Changed on Jul 23, 2020 at 8:58:17 AM by Sebastian Huber <sebastian.huber@…>

In 34e4df55/rtems:

libtest: Split POSIX Keys support

Update #3199.

comment:46 Changed on Jul 23, 2020 at 8:58:20 AM by Sebastian Huber <sebastian.huber@…>

In 7781404/rtems:

libtest: Add T_stop()

Update #3199.

comment:47 Changed on Jul 23, 2020 at 8:58:24 AM by Sebastian Huber <sebastian.huber@…>

In 35d9af69/rtems:

libtest: Add T_CHECK_FMT

Rename internal function T_check_true() to T_check() and use the new
flag T_CHECK_FMT to indicate if a format string is present. This is a
preparation step to make the format string optional.

Make the check context the first parameter.

The API remains the same.

Update #3199.

comment:48 Changed on Jul 23, 2020 at 8:58:27 AM by Sebastian Huber <sebastian.huber@…>

In d702c9f4/rtems:

libtest: Make check message optional

This macro magic is in line with C11 and C++11, but limits the maximum
count of arguments.

Update #3199.

comment:49 Changed on Jul 23, 2020 at 8:58:30 AM by Sebastian Huber <sebastian.huber@…>

In 467ef5b4/rtems:

libtest: Add T_unreachable()

Update #3199.

comment:50 Changed on Jul 23, 2020 at 8:58:34 AM by Sebastian Huber <sebastian.huber@…>

In dddc9a58/rtems:

libtest: Add quiet assert NULL pointer checks

Update #3199.

comment:51 Changed on Jul 23, 2020 at 8:58:37 AM by Sebastian Huber <sebastian.huber@…>

In c9d2405/rtems:

libtest: Add rtems_test_run()

Update #3199.

comment:52 Changed on Jul 23, 2020 at 8:58:40 AM by Sebastian Huber <sebastian.huber@…>

In cc3fd8fc/rtems:

libtest: Add T_interrupt_test()

Update #3199.

comment:53 Changed on Aug 6, 2020 at 5:16:12 PM by Sebastian Huber <sebastian.huber@…>

In 32f1f747/rtems:

libtest: Fix T_interrupt_test() in SMP configs

Update #3199.

comment:54 Changed on Aug 7, 2020 at 4:10:19 PM by Sebastian Huber <sebastian.huber@…>

In f933b65/rtems:

libtest: Improve T_check_task_context

Update #3199.

comment:55 Changed on Aug 10, 2020 at 9:50:29 AM by Sebastian Huber <sebastian.huber@…>

In 20c79bf/rtems:

libtest: Constify

Update #3199.

comment:56 Changed on Aug 11, 2020 at 6:04:20 AM by Sebastian Huber <sebastian.huber@…>

In c1354f0/rtems:

libtest: Add T_thread_switch_record()

Add support to record thread switch events. This can be used to check
that a blocking operation results in the expected sequence of thread
switches.

Update #3199.

comment:57 Changed on Aug 18, 2020 at 5:15:18 AM by Sebastian Huber <sebastian.huber@…>

In 5a8114c6/rtems:

libtest: Change fixture scope method

Return the produced character count. There is no need for a NUL
termination.

Update #3199.

comment:58 Changed on Aug 18, 2020 at 5:15:21 AM by Sebastian Huber <sebastian.huber@…>

In e67eff2/rtems:

libtest: Add output buffer drain and fill

Update #3199.

comment:59 Changed on Aug 18, 2020 at 5:15:25 AM by Sebastian Huber <sebastian.huber@…>

In a7af34d0/rtems:

libtest: Add T_do_is_runner()

Update #3199.

comment:60 Changed on Aug 18, 2020 at 5:15:28 AM by Sebastian Huber <sebastian.huber@…>

In 5d614fd/rtems:

libtest: Add T_puts()

Update #3199.

comment:61 Changed on Aug 18, 2020 at 5:15:31 AM by Sebastian Huber <sebastian.huber@…>

In cbc1ba3/rtems:

libtest: Use line buffer in T_check()

Update #3199.

comment:62 Changed on Aug 18, 2020 at 5:15:34 AM by Sebastian Huber <sebastian.huber@…>

In 33eb113/rtems:

libtest: Add T_check_steps()

Update #3199.

comment:63 Changed on Aug 18, 2020 at 5:15:37 AM by Sebastian Huber <sebastian.huber@…>

In 5383d4db/rtems:

libtest: Add fixture steps

Support a new test plan for each nested fixture.

Update #3199.

comment:64 Changed on Aug 18, 2020 at 5:15:41 AM by Sebastian Huber <sebastian.huber@…>

In 72960bc7/rtems:

libtest: Change T_step() and T_assert_step()

Normally, the expected test step must be a compile time constant. Allow
variable expected test steps for the T_step() and T_assert_step(). This
can be used for parameterized test loops with individual fixtures.

Remove the ability to use custom failure messages due to some
implementation constraints.

Update #3199.

comment:65 Changed on Aug 18, 2020 at 5:15:44 AM by Sebastian Huber <sebastian.huber@…>

In 13acd90/rtems:

libtest: Use a destructor

Do not set up a new test steps environment.

Update #3199.

comment:66 Changed on Aug 18, 2020 at 5:15:47 AM by Sebastian Huber <sebastian.huber@…>

In c7289484/rtems:

libtest: Add T_push_plan() and T_pop_plan()

Update #3199.

comment:67 Changed on Aug 31, 2020 at 2:21:37 PM by Sebastian Huber <sebastian.huber@…>

In d556af36/rtems:

bsps: Always install IPI in SMP configs

The inter-processor interrupt (IPI) may be used to process per-CPU jobs.
See for example the blocked handler in T_interrupt_test().

Update #3199.

comment:68 Changed on Sep 17, 2020 at 4:09:12 PM by Sebastian Huber <sebastian.huber@…>

In 98d2adb/rtems:

libtest: Fix T_thread_switch_record()

If RTEMS_DEBUG is not defined, then we have to explicitly set the node
off the chain.

Update #3199.

comment:69 Changed on Nov 19, 2020 at 7:39:42 AM by Sebastian Huber <sebastian.huber@…>

In c542345/rtems:

libtest: Simplify "Load" environment reporting

Report all runtime measurement environments with a name only and encode
the worker count of the "Load" environment in the name.

Update #3199.

comment:70 Changed on Nov 19, 2020 at 7:59:18 AM by Sebastian Huber <sebastian.huber@…>

In cdbd72d/rtems-docs:

eng: Rename ValidCache? environment in FullCache?

This name better reflects the execution envirnoment in which the cache
is fully loaded with valid data unrelated to the body request handler.

Update #3199.

comment:71 Changed on Nov 19, 2020 at 7:59:20 AM by Sebastian Huber <sebastian.huber@…>

In 0454ad7/rtems-docs:

eng: Simplify "Load" environment reporting

Report all runtime measurement environments with a name only and encode
the worker count of the "Load" environment in the name.

Update #3199.

comment:72 Changed on Nov 24, 2020 at 6:40:45 AM by Sebastian Huber <sebastian.huber@…>

In aa1c6dd/rtems:

libtest: Fix undefined setjmp() behaviour

Bug was introduced by 78baeb757957fa0807c30e6c4d21ae99c9639e6a.

Update #3199.

comment:73 Changed on Jun 18, 2021 at 9:24:45 AM by Sebastian Huber

Keywords: qualification added
Note: See TracTickets for help on using tickets.