#3199 closed enhancement (fixed)

New test framework

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: normal Milestone: 6.1
Component: lib Version: 5
Severity: normal Keywords: qualification
Cc: Blocked By:
Blocking: #3716

Description

Requirements:

  • Support for RTEMS and general POSIX systems such as Linux or FreeBSD
  • Test output functions
    • printf() like
    • No floating point support by default
    • Optional floating point support
    • Non-blocking
    • Arbitrary context, e.g. during system start, before device drivers, in interrupt context
  • Automatic test case registrations (constructors, linker set)
  • Standard test case output, e.g. begin/end of test case message
  • Ability to sort test cases at run-time
  • Safe assert functions, e.g. assert(a == b) vs. assert(a = b) vs. assert_eq(a, b)
  • Easy to write, e.g. avoid long namespace prefix rtems_test_*
  • Support multiple threads
  • No dependency on C++
  • Context validation, context invariant validation (e.g. thread dispatch disable level, interrupt state, heap state)
  • Test pattern support (e.g. spintercritical*, parallel tests on SMP)

No ready to use framework exists which fulfils these requirements. We have to write something for RTEMS.

API proposal (header file <T.h>):

T_eq(a, b)
T_gt(a, b)
T_printf(fmt, ...)
T_output_printf(putchar, fmt, ...)
T_output_string(putchar, s, n)
T_output_add_format_handler(handler, arg)

Example for T_output_add_format():

#include <stdio.h>
#include <string.h>

typedef void (*T_putchar)(char c);

typedef size_t (*T_format_handler)(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out);

int T_printf(const char *fmt, ...);

void T_output_printf(T_putchar putchar, const char *fmt, ...);

void T_output_string(T_putchar putchar, const char *s, size_t n);

void T_output_add_format_handler(T_format_handler handler, void *arg);

size_t T_output_float_handler(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out)
{
  char buf[32];

  if (strncmp(fmt, "%f", 2) != 0) {
    return 0;
  }

  *out = snprintf(buf, sizeof(buf), "%f", s);

  if (*out > 0) {
    T_output_string(putchar, s, (size_t) *out);
  }

  return 2;
}

void f(void)
{
  T_output_add_format_handler(T_output_float_handler, NULL);
}

Attachments (2)

t.tar.gz (26.5 KB) - added by Sebastian Huber on 11/23/18 at 10:26:10.
The T Test Framework
contributor.pdf (188.8 KB) - added by Sebastian Huber on 12/06/18 at 14:47:59.
Draft v4

Download all attachments as: .zip

Change History (76)

comment:2 Changed on 10/28/17 at 12:08:04 by Sebastian Huber <sebastian.huber@…>

In f703e7f/rtems:

tests: Move rtems_test_printer definition

Statically initialize it to use printk().

Update #3170.
Update #3199.

comment:3 Changed on 10/28/17 at 12:08:16 by Sebastian Huber <sebastian.huber@…>

In 7bec7f27/rtems:

rtems: Add rtems_print_printer_fprintf_putc()

Update #3170.
Update #3199.

comment:4 Changed on 10/28/17 at 12:08:27 by Sebastian Huber <sebastian.huber@…>

In 73d892d8/rtems:

tests: Use rtems_test_printer

Update #3170.
Update #3199.

comment:5 Changed on 10/28/17 at 12:08:39 by Sebastian Huber <sebastian.huber@…>

In 46ddc3c5/rtems:

tests: Use rtems_print_printer_fprintf_putc()

Use rtems_print_printer_fprintf_putc() instead of
rtems_print_printer_printf() to output via rtems_putc().

Update #3170.
Update #3199.

comment:6 Changed on 10/28/17 at 12:08:51 by Sebastian Huber <sebastian.huber@…>

In 7e10291/rtems:

tests: Use rtems_test_printer in general

Update #3170.
Update #3199.

comment:7 Changed on 10/28/17 at 12:09:02 by Sebastian Huber <sebastian.huber@…>

In acc9d064/rtems:

tests: Remove obsolete TESTS_USE_PRINTK

Update #3170.
Update #3199.

comment:8 Changed on 10/28/17 at 12:09:15 by Sebastian Huber <sebastian.huber@…>

In af43554/rtems:

tests: Remove TEST_INIT

The TEST_EXTERN is a used only by the system.h style tests and they use
CONFIGURE_INIT appropriately.

Update #3170.
Update #3199.

comment:9 Changed on 10/30/17 at 05:31:17 by Chris Johns <chrisj@…>

In 2126438a/rtems:

testsuite: Add bspIo for a local printk.

Update #3170.
Update #3199.

comment:10 Changed on 11/02/17 at 13:25:12 by Sebastian Huber <sebastian.huber@…>

In 0d796d6/rtems:

tests: Delete obsolete TESTS_USE_PRINTF

Update #3170.
Update #3199.

comment:11 Changed on 11/02/17 at 13:26:51 by Sebastian Huber <sebastian.huber@…>

In 8c1f4064/rtems:

tests: Use printf() instead of fprintf()

Update #3170.
Update #3199.

comment:12 Changed on 11/06/17 at 06:29:13 by Sebastian Huber <sebastian.huber@…>

In 1082798/rtems:

score: Add _IO_Printf() and _IO_Vprintf()

The previous vprintk() implementation had a questionable licence header,
lacks support for the 'z' and 'j' format specifiers, is not robust
against invalid format specifiers, uses a global variable for output.
Replace it with a stripped down version of the FreeBSD kernel kvprintf()
function.

The new implementation allows a low overhead rtems_snprintf() if
necessary.

Update #3199.
Close #3216.

comment:13 Changed on 11/06/17 at 06:29:25 by Sebastian Huber <sebastian.huber@…>

In ac28f15/rtems:

Add simple console driver

Update #3170.
Update #3199.

comment:14 Changed on 11/06/17 at 06:29:41 by Sebastian Huber <sebastian.huber@…>

In c4b8b147/rtems:

tests: Use simple console driver

Update #3170.
Update #3199.

comment:15 Changed on 11/07/17 at 06:09:15 by Sebastian Huber <sebastian.huber@…>

In 7b00c2fa/rtems:

tests: Use <tmacros.h> in all tests

Update #3170.
Update #3199.

comment:16 Changed on 11/07/17 at 07:32:57 by Sebastian Huber <sebastian.huber@…>

In 32ceb38/rtems:

tests: Use <tmacros.h>

Update #3170.
Update #3199.

comment:17 Changed on 11/10/17 at 02:42:58 by Chris Johns <chrisj@…>

In 6f3fb8a/rtems:

cpukit: Add a Version API.

Provide functions to get the version string, major, minor and revision
numbers and the version control identifer that is a unique tag for
the version control system.

Update #3199.

comment:18 Changed on 11/11/17 at 05:24:06 by Chris Johns <chrisj@…>

In e6df806/rtems:

tests: Use ld to map (wrap) printf, puts and putchar to tester functions.

  • Remove the macro defines and the need for tmacro.h by remapping the symbols using ld's wrap option.
  • Remove FLUSH_OUTPUT, it was empty.
  • Move rtems_test_exit to libmisc/testsupport as a function.

Update #3199.

comment:19 Changed on 11/15/17 at 07:10:55 by Sebastian Huber <sebastian.huber@…>

In 62119d2/rtems:

dl01, dl02, dl05: Fix unresolved printf symbol

The link time wrap of printf leads to unresolved symbols in the loadable
modules. This resulted in infinite loops and test timeouts. Use
rtems_printf() for output.

Update #3199.

comment:20 Changed on 11/17/17 at 05:55:31 by Sebastian Huber <sebastian.huber@…>

In 727cf48/rtems:

sptests/spversion01: Simplify configuration

Update #3199.

comment:21 Changed on 02/05/18 at 09:48:37 by Sebastian Huber <sebastian.huber@…>

In d078405/rtems-docs:

CONFIGURE_APPLICATION_NEEDS_SIMPLE_CONSOLE_DRIVER

Close #3170.
Update #3199.

Changed on 11/23/18 at 10:26:10 by Sebastian Huber

Attachment: t.tar.gz added

The T Test Framework

Changed on 12/06/18 at 14:47:59 by Sebastian Huber

Attachment: contributor.pdf added

Draft v4

comment:22 Changed on 03/04/19 at 13:03:31 by Sebastian Huber

Blocking: 3716 added
Status: assignedaccepted

comment:23 Changed on 03/27/19 at 06:29:28 by Sebastian Huber <sebastian.huber@…>

In 6fe01e4b/rtems:

build: Move test support to librtemstest.a

One reason to move the test support into a dedicated library are the
standard output wrap_*() functions. They may conflict with
application level wrappers.

Update #3199.

comment:24 Changed on 03/27/19 at 06:29:32 by Sebastian Huber <sebastian.huber@…>

In cfcc2cbf/rtems:

Add RTEMS Test Framework

Update #3199.

comment:25 Changed on 03/27/19 at 06:29:37 by Sebastian Huber <sebastian.huber@…>

In cbfc3415/rtems:

ttest01: New test

This is an example test using the RTEMS Test Framework. It tests also
the framework itself.

Add T_FILE_NAME command line define to get rid of the full file path.
This is important to reduce the read-only data of test files and make
them build system independent.

Update #3199.

comment:26 Changed on 03/27/19 at 06:34:23 by Sebastian Huber <sebastian.huber@…>

In c2e582d/rtems-docs:

eng: Add software test framework chapter

Update #3199.

comment:27 Changed on 10/11/19 at 09:02:08 by Sebastian Huber <sebastian.huber@…>

In f88025a/rtems:

ttest01: Adjust SPDX-License-Identifier

Update #3199.

comment:28 Changed on 10/11/19 at 09:02:13 by Sebastian Huber <sebastian.huber@…>

In b406d071/rtems:

libtest: Do all output in test runner

This ensures that lines are output atomically if they are produced by
different other contexts, e.g. interrupts, other processors, other
threads.

Update #3199.

comment:29 Changed on 10/11/19 at 09:02:18 by Sebastian Huber <sebastian.huber@…>

In feb27f9/rtems:

ttest01: Add more test cases

Update #3199.

comment:30 Changed on 10/11/19 at 09:02:26 by Sebastian Huber <sebastian.huber@…>

In b5e61f95/rtems:

libtest: Add more action events

This allows more control over the initialization and finalization run.

Update #3199.

comment:31 Changed on 10/11/19 at 09:02:32 by Sebastian Huber <sebastian.huber@…>

In e71f0a5/rtems:

ttest01: Check init/final run output

Update #3199.

comment:32 Changed on 02/10/20 at 08:03:35 by Sebastian Huber <sebastian.huber@…>

In 45a0f65/rtems-docs:

eng: Document test framework formatted output

Update #3199.

comment:33 Changed on 02/10/20 at 08:03:37 by Sebastian Huber <sebastian.huber@…>

In b6c61e3/rtems-docs:

eng: Mention test framework buffer configuration

Update #3199.

comment:34 Changed on 02/10/20 at 12:49:18 by Sebastian Huber <sebastian.huber@…>

In 76b3aea/rtems-docs:

eng: Grammar fix

Update #3199.

comment:35 Changed on 07/17/20 at 09:23:51 by Sebastian Huber

Milestone: Indefinite6.1

comment:36 Changed on 07/23/20 at 08:06:12 by Sebastian Huber <sebastian.huber@…>

In 21fa28c/rtems-docs:

eng: Update test framework chapter

Document the dynamic text fixtures, utility functions, and the interrupt
test support. Reorder some sections and reword some paragraphs based on
review comments.

Update #3199.

comment:37 Changed on 07/23/20 at 08:57:50 by Sebastian Huber <sebastian.huber@…>

In 9de8d61/rtems:

libtest: <rtems/test.h> to <rtems/test-info.h>

Rename this header file to later move <t.h> to <rtems/test.h>. The main
feature provided by <rtems/test-info.h> is the output of standard test
information which is consumed by the RTEMS Tester.

Update #3199.

comment:38 Changed on 07/23/20 at 08:57:54 by Sebastian Huber <sebastian.huber@…>

In 361404e8/rtems:

libtest: Move <t.h> to <rtems/test.h>

Update #3199.

comment:39 Changed on 07/23/20 at 08:57:57 by Sebastian Huber <sebastian.huber@…>

In e3e3b871/rtems:

libtest: Add T_busy()

Update #3199.

comment:40 Changed on 07/23/20 at 08:58:00 by Sebastian Huber <sebastian.huber@…>

In af92665/rtems:

libtest: Add T_get_one_clock_tick_busy()

Update #3199.

comment:41 Changed on 07/23/20 at 08:58:04 by Sebastian Huber <sebastian.huber@…>

In c081c68/rtems:

libtest: Add T_make_runner()

Update #3199.

comment:42 Changed on 07/23/20 at 08:58:07 by Sebastian Huber <sebastian.huber@…>

In 6b27e32/rtems:

libtest: Support custom scope messages via fixture

Update #3199.

comment:43 Changed on 07/23/20 at 08:58:10 by Sebastian Huber <sebastian.huber@…>

In cb3c6bdc/rtems:

libtest: Add push/pop fixture support

Update #3199.

comment:44 Changed on 07/23/20 at 08:58:14 by Sebastian Huber <sebastian.huber@…>

In 63e4278/rtems:

libtest: Add T_get_scope()

Update #3199.

comment:45 Changed on 07/23/20 at 08:58:17 by Sebastian Huber <sebastian.huber@…>

In 34e4df55/rtems:

libtest: Split POSIX Keys support

Update #3199.

comment:46 Changed on 07/23/20 at 08:58:20 by Sebastian Huber <sebastian.huber@…>

In 7781404/rtems:

libtest: Add T_stop()

Update #3199.

comment:47 Changed on 07/23/20 at 08:58:24 by Sebastian Huber <sebastian.huber@…>

In 35d9af69/rtems:

libtest: Add T_CHECK_FMT

Rename internal function T_check_true() to T_check() and use the new
flag T_CHECK_FMT to indicate if a format string is present. This is a
preparation step to make the format string optional.

Make the check context the first parameter.

The API remains the same.

Update #3199.

comment:48 Changed on 07/23/20 at 08:58:27 by Sebastian Huber <sebastian.huber@…>

In d702c9f4/rtems:

libtest: Make check message optional

This macro magic is in line with C11 and C++11, but limits the maximum
count of arguments.

Update #3199.

comment:49 Changed on 07/23/20 at 08:58:30 by Sebastian Huber <sebastian.huber@…>

In 467ef5b4/rtems:

libtest: Add T_unreachable()

Update #3199.

comment:50 Changed on 07/23/20 at 08:58:34 by Sebastian Huber <sebastian.huber@…>

In dddc9a58/rtems:

libtest: Add quiet assert NULL pointer checks

Update #3199.

comment:51 Changed on 07/23/20 at 08:58:37 by Sebastian Huber <sebastian.huber@…>

In c9d2405/rtems:

libtest: Add rtems_test_run()

Update #3199.

comment:52 Changed on 07/23/20 at 08:58:40 by Sebastian Huber <sebastian.huber@…>

In cc3fd8fc/rtems:

libtest: Add T_interrupt_test()

Update #3199.

comment:53 Changed on 08/06/20 at 17:16:12 by Sebastian Huber <sebastian.huber@…>

In 32f1f747/rtems:

libtest: Fix T_interrupt_test() in SMP configs

Update #3199.

comment:54 Changed on 08/07/20 at 16:10:19 by Sebastian Huber <sebastian.huber@…>

In f933b65/rtems:

libtest: Improve T_check_task_context

Update #3199.

comment:55 Changed on 08/10/20 at 09:50:29 by Sebastian Huber <sebastian.huber@…>

In 20c79bf/rtems:

libtest: Constify

Update #3199.

comment:56 Changed on 08/11/20 at 06:04:20 by Sebastian Huber <sebastian.huber@…>

In c1354f0/rtems:

libtest: Add T_thread_switch_record()

Add support to record thread switch events. This can be used to check
that a blocking operation results in the expected sequence of thread
switches.

Update #3199.

comment:57 Changed on 08/18/20 at 05:15:18 by Sebastian Huber <sebastian.huber@…>

In 5a8114c6/rtems:

libtest: Change fixture scope method

Return the produced character count. There is no need for a NUL
termination.

Update #3199.

comment:58 Changed on 08/18/20 at 05:15:21 by Sebastian Huber <sebastian.huber@…>

In e67eff2/rtems:

libtest: Add output buffer drain and fill

Update #3199.

comment:59 Changed on 08/18/20 at 05:15:25 by Sebastian Huber <sebastian.huber@…>

In a7af34d0/rtems:

libtest: Add T_do_is_runner()

Update #3199.

comment:60 Changed on 08/18/20 at 05:15:28 by Sebastian Huber <sebastian.huber@…>

In 5d614fd/rtems:

libtest: Add T_puts()

Update #3199.

comment:61 Changed on 08/18/20 at 05:15:31 by Sebastian Huber <sebastian.huber@…>

In cbc1ba3/rtems:

libtest: Use line buffer in T_check()

Update #3199.

comment:62 Changed on 08/18/20 at 05:15:34 by Sebastian Huber <sebastian.huber@…>

In 33eb113/rtems:

libtest: Add T_check_steps()

Update #3199.

comment:63 Changed on 08/18/20 at 05:15:37 by Sebastian Huber <sebastian.huber@…>

In 5383d4db/rtems:

libtest: Add fixture steps

Support a new test plan for each nested fixture.

Update #3199.

comment:64 Changed on 08/18/20 at 05:15:41 by Sebastian Huber <sebastian.huber@…>

In 72960bc7/rtems:

libtest: Change T_step() and T_assert_step()

Normally, the expected test step must be a compile time constant. Allow
variable expected test steps for the T_step() and T_assert_step(). This
can be used for parameterized test loops with individual fixtures.

Remove the ability to use custom failure messages due to some
implementation constraints.

Update #3199.

comment:65 Changed on 08/18/20 at 05:15:44 by Sebastian Huber <sebastian.huber@…>

In 13acd90/rtems:

libtest: Use a destructor

Do not set up a new test steps environment.

Update #3199.

comment:66 Changed on 08/18/20 at 05:15:47 by Sebastian Huber <sebastian.huber@…>

In c7289484/rtems:

libtest: Add T_push_plan() and T_pop_plan()

Update #3199.

comment:67 Changed on 08/31/20 at 14:21:37 by Sebastian Huber <sebastian.huber@…>

In d556af36/rtems:

bsps: Always install IPI in SMP configs

The inter-processor interrupt (IPI) may be used to process per-CPU jobs.
See for example the blocked handler in T_interrupt_test().

Update #3199.

comment:68 Changed on 09/17/20 at 16:09:12 by Sebastian Huber <sebastian.huber@…>

In 98d2adb/rtems:

libtest: Fix T_thread_switch_record()

If RTEMS_DEBUG is not defined, then we have to explicitly set the node
off the chain.

Update #3199.

comment:69 Changed on 11/19/20 at 07:39:42 by Sebastian Huber <sebastian.huber@…>

In c542345/rtems:

libtest: Simplify "Load" environment reporting

Report all runtime measurement environments with a name only and encode
the worker count of the "Load" environment in the name.

Update #3199.

comment:70 Changed on 11/19/20 at 07:59:18 by Sebastian Huber <sebastian.huber@…>

In cdbd72d/rtems-docs:

eng: Rename ValidCache? environment in FullCache?

This name better reflects the execution envirnoment in which the cache
is fully loaded with valid data unrelated to the body request handler.

Update #3199.

comment:71 Changed on 11/19/20 at 07:59:20 by Sebastian Huber <sebastian.huber@…>

In 0454ad7/rtems-docs:

eng: Simplify "Load" environment reporting

Report all runtime measurement environments with a name only and encode
the worker count of the "Load" environment in the name.

Update #3199.

comment:72 Changed on 11/24/20 at 06:40:45 by Sebastian Huber <sebastian.huber@…>

In aa1c6dd/rtems:

libtest: Fix undefined setjmp() behaviour

Bug was introduced by 78baeb757957fa0807c30e6c4d21ae99c9639e6a.

Update #3199.

comment:73 Changed on 06/18/21 at 09:24:45 by Sebastian Huber

Keywords: qualification added

comment:74 Changed on 09/29/21 at 13:25:31 by Sebastian Huber

Resolution: fixed
Status: acceptedclosed

The test framework is fully implemented.

Note: See TracTickets for help on using tickets.