#3199 accepted enhancement

New test framework

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: normal Milestone: Indefinite
Component: lib Version: 5
Severity: normal Keywords:
Cc: Blocked By:
Blocking: #3716

Description

Requirements:

  • Support for RTEMS and general POSIX systems such as Linux or FreeBSD
  • Test output functions
    • printf() like
    • No floating point support by default
    • Optional floating point support
    • Non-blocking
    • Arbitrary context, e.g. during system start, before device drivers, in interrupt context
  • Automatic test case registrations (constructors, linker set)
  • Standard test case output, e.g. begin/end of test case message
  • Ability to sort test cases at run-time
  • Safe assert functions, e.g. assert(a == b) vs. assert(a = b) vs. assert_eq(a, b)
  • Easy to write, e.g. avoid long namespace prefix rtems_test_*
  • Support multiple threads
  • No dependency on C++
  • Context validation, context invariant validation (e.g. thread dispatch disable level, interrupt state, heap state)
  • Test pattern support (e.g. spintercritical*, parallel tests on SMP)

No ready to use framework exists which fulfils these requirements. We have to write something for RTEMS.

API proposal (header file <T.h>):

T_eq(a, b)
T_gt(a, b)
T_printf(fmt, ...)
T_output_printf(putchar, fmt, ...)
T_output_string(putchar, s, n)
T_output_add_format_handler(handler, arg)

Example for T_output_add_format():

#include <stdio.h>
#include <string.h>

typedef void (*T_putchar)(char c);

typedef size_t (*T_format_handler)(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out);

int T_printf(const char *fmt, ...);

void T_output_printf(T_putchar putchar, const char *fmt, ...);

void T_output_string(T_putchar putchar, const char *s, size_t n);

void T_output_add_format_handler(T_format_handler handler, void *arg);

size_t T_output_float_handler(void *arg, T_putchar putchar, const char *fmt, const char *s, int *out)
{
  char buf[32];

  if (strncmp(fmt, "%f", 2) != 0) {
    return 0;
  }

  *out = snprintf(buf, sizeof(buf), "%f", s);

  if (*out > 0) {
    T_output_string(putchar, s, (size_t) *out);
  }

  return 2;
}

void f(void)
{
  T_output_add_format_handler(T_output_float_handler, NULL);
}

Attachments (2)

t.tar.gz (26.5 KB) - added by Sebastian Huber on Nov 23, 2018 at 10:26:10 AM.
The T Test Framework
contributor.pdf (188.8 KB) - added by Sebastian Huber on Dec 6, 2018 at 2:47:59 PM.
Draft v4

Download all attachments as: .zip

Change History (28)

comment:2 Changed on Oct 28, 2017 at 12:08:04 PM by Sebastian Huber <sebastian.huber@…>

In f703e7f/rtems:

tests: Move rtems_test_printer definition

Statically initialize it to use printk().

Update #3170.
Update #3199.

comment:3 Changed on Oct 28, 2017 at 12:08:16 PM by Sebastian Huber <sebastian.huber@…>

In 7bec7f27/rtems:

rtems: Add rtems_print_printer_fprintf_putc()

Update #3170.
Update #3199.

comment:4 Changed on Oct 28, 2017 at 12:08:27 PM by Sebastian Huber <sebastian.huber@…>

In 73d892d8/rtems:

tests: Use rtems_test_printer

Update #3170.
Update #3199.

comment:5 Changed on Oct 28, 2017 at 12:08:39 PM by Sebastian Huber <sebastian.huber@…>

In 46ddc3c5/rtems:

tests: Use rtems_print_printer_fprintf_putc()

Use rtems_print_printer_fprintf_putc() instead of
rtems_print_printer_printf() to output via rtems_putc().

Update #3170.
Update #3199.

comment:6 Changed on Oct 28, 2017 at 12:08:51 PM by Sebastian Huber <sebastian.huber@…>

In 7e10291/rtems:

tests: Use rtems_test_printer in general

Update #3170.
Update #3199.

comment:7 Changed on Oct 28, 2017 at 12:09:02 PM by Sebastian Huber <sebastian.huber@…>

In acc9d064/rtems:

tests: Remove obsolete TESTS_USE_PRINTK

Update #3170.
Update #3199.

comment:8 Changed on Oct 28, 2017 at 12:09:15 PM by Sebastian Huber <sebastian.huber@…>

In af43554/rtems:

tests: Remove TEST_INIT

The TEST_EXTERN is a used only by the system.h style tests and they use
CONFIGURE_INIT appropriately.

Update #3170.
Update #3199.

comment:9 Changed on Oct 30, 2017 at 5:31:17 AM by Chris Johns <chrisj@…>

In 2126438a/rtems:

testsuite: Add bspIo for a local printk.

Update #3170.
Update #3199.

comment:10 Changed on Nov 2, 2017 at 1:25:12 PM by Sebastian Huber <sebastian.huber@…>

In 0d796d6/rtems:

tests: Delete obsolete TESTS_USE_PRINTF

Update #3170.
Update #3199.

comment:11 Changed on Nov 2, 2017 at 1:26:51 PM by Sebastian Huber <sebastian.huber@…>

In 8c1f4064/rtems:

tests: Use printf() instead of fprintf()

Update #3170.
Update #3199.

comment:12 Changed on Nov 6, 2017 at 6:29:13 AM by Sebastian Huber <sebastian.huber@…>

In 1082798/rtems:

score: Add _IO_Printf() and _IO_Vprintf()

The previous vprintk() implementation had a questionable licence header,
lacks support for the 'z' and 'j' format specifiers, is not robust
against invalid format specifiers, uses a global variable for output.
Replace it with a stripped down version of the FreeBSD kernel kvprintf()
function.

The new implementation allows a low overhead rtems_snprintf() if
necessary.

Update #3199.
Close #3216.

comment:13 Changed on Nov 6, 2017 at 6:29:25 AM by Sebastian Huber <sebastian.huber@…>

In ac28f15/rtems:

Add simple console driver

Update #3170.
Update #3199.

comment:14 Changed on Nov 6, 2017 at 6:29:41 AM by Sebastian Huber <sebastian.huber@…>

In c4b8b147/rtems:

tests: Use simple console driver

Update #3170.
Update #3199.

comment:15 Changed on Nov 7, 2017 at 6:09:15 AM by Sebastian Huber <sebastian.huber@…>

In 7b00c2fa/rtems:

tests: Use <tmacros.h> in all tests

Update #3170.
Update #3199.

comment:16 Changed on Nov 7, 2017 at 7:32:57 AM by Sebastian Huber <sebastian.huber@…>

In 32ceb38/rtems:

tests: Use <tmacros.h>

Update #3170.
Update #3199.

comment:17 Changed on Nov 10, 2017 at 2:42:58 AM by Chris Johns <chrisj@…>

In 6f3fb8a/rtems:

cpukit: Add a Version API.

Provide functions to get the version string, major, minor and revision
numbers and the version control identifer that is a unique tag for
the version control system.

Update #3199.

comment:18 Changed on Nov 11, 2017 at 5:24:06 AM by Chris Johns <chrisj@…>

In e6df806/rtems:

tests: Use ld to map (wrap) printf, puts and putchar to tester functions.

  • Remove the macro defines and the need for tmacro.h by remapping the symbols using ld's wrap option.
  • Remove FLUSH_OUTPUT, it was empty.
  • Move rtems_test_exit to libmisc/testsupport as a function.

Update #3199.

comment:19 Changed on Nov 15, 2017 at 7:10:55 AM by Sebastian Huber <sebastian.huber@…>

In 62119d2/rtems:

dl01, dl02, dl05: Fix unresolved printf symbol

The link time wrap of printf leads to unresolved symbols in the loadable
modules. This resulted in infinite loops and test timeouts. Use
rtems_printf() for output.

Update #3199.

comment:20 Changed on Nov 17, 2017 at 5:55:31 AM by Sebastian Huber <sebastian.huber@…>

In 727cf48/rtems:

sptests/spversion01: Simplify configuration

Update #3199.

comment:21 Changed on Feb 5, 2018 at 9:48:37 AM by Sebastian Huber <sebastian.huber@…>

In d078405/rtems-docs:

CONFIGURE_APPLICATION_NEEDS_SIMPLE_CONSOLE_DRIVER

Close #3170.
Update #3199.

Changed on Nov 23, 2018 at 10:26:10 AM by Sebastian Huber

Attachment: t.tar.gz added

The T Test Framework

Changed on Dec 6, 2018 at 2:47:59 PM by Sebastian Huber

Attachment: contributor.pdf added

Draft v4

comment:22 Changed on Mar 4, 2019 at 1:03:31 PM by Sebastian Huber

Blocking: 3716 added
Status: assignedaccepted

comment:23 Changed on Mar 27, 2019 at 6:29:28 AM by Sebastian Huber <sebastian.huber@…>

In 6fe01e4b/rtems:

build: Move test support to librtemstest.a

One reason to move the test support into a dedicated library are the
standard output wrap_*() functions. They may conflict with
application level wrappers.

Update #3199.

comment:24 Changed on Mar 27, 2019 at 6:29:32 AM by Sebastian Huber <sebastian.huber@…>

In cfcc2cbf/rtems:

Add RTEMS Test Framework

Update #3199.

comment:25 Changed on Mar 27, 2019 at 6:29:37 AM by Sebastian Huber <sebastian.huber@…>

In cbfc3415/rtems:

ttest01: New test

This is an example test using the RTEMS Test Framework. It tests also
the framework itself.

Add T_FILE_NAME command line define to get rid of the full file path.
This is important to reduce the read-only data of test files and make
them build system independent.

Update #3199.

comment:26 Changed on Mar 27, 2019 at 6:34:23 AM by Sebastian Huber <sebastian.huber@…>

In c2e582d/rtems-docs:

eng: Add software test framework chapter

Update #3199.

Note: See TracTickets for help on using tickets.