Changeset 21fa28c in rtems-docs


Ignore:
Timestamp:
Jul 21, 2020, 2:38:49 PM (3 weeks ago)
Author:
Sebastian Huber <sebastian.huber@…>
Branches:
master
Children:
7ee5a7b
Parents:
31b1d88
git-author:
Sebastian Huber <sebastian.huber@…> (07/21/20 14:38:49)
git-committer:
Sebastian Huber <sebastian.huber@…> (07/23/20 08:04:59)
Message:

eng: Update test framework chapter

Document the dynamic text fixtures, utility functions, and the interrupt
test support. Reorder some sections and reword some paragraphs based on
review comments.

Update #3199.

Files:
3 added
2 edited

Legend:

Unmodified
Added
Removed
  • eng/index.rst

    r31b1d88 r21fa28c  
    1212.. topic:: Copyrights and License
    1313
    14     | |copy| 2018, 2019 embedded brains GmbH
    15     | |copy| 2018, 2019 Sebastian Huber
     14    | |copy| 2018, 2020 embedded brains GmbH
     15    | |copy| 2018, 2020 Sebastian Huber
    1616    | |copy| 1988, 2015 On-Line Applications Research Corporation (OAR)
    1717
  • eng/test-framework.rst

    r31b1d88 r21fa28c  
    11.. SPDX-License-Identifier: CC-BY-SA-4.0
    22
    3 .. Copyright (C) 2018, 2019 embedded brains GmbH
    4 .. Copyright (C) 2018, 2019 Sebastian Huber
     3.. Copyright (C) 2018, 2020 embedded brains GmbH
     4.. Copyright (C) 2018, 2020 Sebastian Huber
    55
    66Software Test Framework
     
    1616
    1717* Implemented in standard C11
     18
     19* Tests can be written in C or C++
    1820
    1921* Runs on at least FreeBSD, MSYS2, Linux and RTEMS
     
    6870of a test action meets its expectation.  A `test action` is a program sequence
    6971with an observable outcome, for example a function invocation with a return
    70 status.  If the test action outcome is all right, then the test check passes,
    71 otherwise the test check fails.  The test check failures of a test case are
    72 summed up.  A test case passes, if the failure count of this test case is zero,
    73 otherwise the test case fails.  The test suite passes if all test cases pass,
    74 otherwise it fails.
     72status.  If a test action produces the expected outcome as determined by the
     73corresponding test check, then this test check passes, otherwise this test
     74check fails.  The test check failures of a test case are summed up.  A test
     75case passes, if the failure count of this test case is zero, otherwise the test
     76case fails.  The test suite passes if all test cases pass, otherwise it fails.
    7577
    7678Test Cases
     
    9092unique within the test suite.  Just link modules with test cases to the test
    9193runner to form a test suite.  The test cases are automatically registered via
    92 static constructors.
     94static C constructors.
    9395
    9496.. code-block:: c
     
    145147The test case `name` must be a valid C designator.  The test case names must be
    146148unique within the test suite.  The `fixture` must point to a statically
    147 initialized read-only object of type `T_fixture`.  The test fixture
    148 provides methods to setup, stop and tear down a test case.  A context is passed
    149 to the methods.  The initial context is defined by the read-only fixture
    150 object.  The context can be obtained by the `T_fixture_context()`
    151 function.  It can be set within the scope of one test case by the
    152 `T_set_fixture_context()` function.  This can be used for example to
    153 dynamically allocate a test environment in the setup method.
     149initialized read-only object of type `T_fixture`.
     150
     151.. code-block:: c
     152
     153    typedef struct T_fixture {
     154        void (*setup)(void *context);
     155        void (*stop)(void *context);
     156        void (*teardown)(void *context);
     157        void (*scope)(void *context, char *buffer, size_t size);
     158        void *initial_context;
     159    } T_fixture;
     160
     161The test fixture provides methods to setup, stop, and teardown a test case as
     162well as the scope for log messages.  A context is passed to each of the
     163methods.  The initial context is defined by the read-only fixture object.  The
     164context can be obtained by the `T_fixture_context()` function.  The context can
     165be changed within the scope of one test case by the `T_set_fixture_context()`
     166function.  The next test case execution using the same fixture will start again
     167with the initial context defined by the read-only fixture object.  Setting the
     168context can be used for example to dynamically allocate a test environment in
     169the setup method.
     170
     171The test case fixtures of a test case are organized as a stack.  Fixtures can
     172be dynamically added to a test case and removed from a test case via the
     173`T_push_fixture()` and `T_pop_fixture()` functions.
     174
     175.. code-block:: c
     176
     177    void *T_push_fixture(T_fixture_node *node, const T_fixture *fixture);
     178
     179    void T_pop_fixture(void);
     180
     181The `T_push_fixture()` function needs an uninitialized fixture node which must
     182exist until `T_pop_fixture()` is called.  It returns the initial context of the
     183fixture.  At the end of a test case all pushed fixtures are popped
     184automatically.  A call of `T_pop_fixture()` invokes the teardown method of the
     185fixture and must correspond to a previous call to `T_push_fixture()`.
    154186
    155187.. code-block:: c
     
    238270------------------
    239271
    240 Each non-quiet test check fetches and increments the test step counter
    241 atomically.  For each test case execution the planned steps can be specified
    242 with the `T_plan()` function.
     272A non-quiet test check fetches and increments the test step counter atomically.
     273For each test case execution the planned steps can be specified with the
     274`T_plan()` function.
    243275
    244276.. code-block:: c
     
    319351-----------------------------
    320352
    321 The framework can check if various resources are leaked during a test case
     353The framework can check if various resources have leaked during a test case
    322354execution.  The resource checkers are specified by the test run configuration.
    323355On RTEMS, checks for the following resources are available
     
    453485---------------------
    454486
    455 You can add test case destructors with `T_add_destructor()`.  They are called
    456 automatically at the test case end before the resource accounting takes place.
    457 Optionally, a registered destructor can be removed before the test case end
    458 with `T_remove_destructor()`.  The `T_destructor` structure of a destructor
    459 must exist after the return from the test case body.  Do not use stack memory
    460 or dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for
    461 the `T_destructor` structure.
     487You can add test case destructors with `T_add_destructor()`.  The destructors
     488are called automatically at the test case end before the resource accounting
     489takes place.  Optionally, a registered destructor can be removed before the
     490test case end with `T_remove_destructor()`.  The `T_destructor` structure of a
     491destructor must exist after the return from the test case body.  It is
     492recommended to use statically allocated memory.  Do not use stack memory or
     493dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for the
     494`T_destructor` structure.
    462495
    463496.. code-block:: c
     
    498531-----------
    499532
    500 A `test check` determines if the actual value presented to the test check meets
    501 its expectation.  The actual value should represent the outcome of a test
    502 action.  If the actual value is all right, then the test check passes,
    503 otherwise the test check fails.  A failed test check does not stop the test
    504 case execution immediately unless the `T_assert_*()` test variant is used.
    505 Each test check increments the test step counter unless the `T_quiet_*()` test
    506 variant is used.  The test step counter is initialized to zero before the test
    507 case begins to execute.  The `T_step_*(step, ...)` test check variants verify
    508 that the test step counter is equal to the planned test step value, otherwise
    509 the test check fails.
     533A `test check` determines if the actual value presented to the test check has
     534the expected properties.  The actual value should represent the outcome of a
     535test action.  If a test action produces the expected outcome as determined by
     536the corresponding test check, then this test check passes, otherwise this test
     537check fails.  A failed test check does not stop the test case execution
     538immediately unless the `T_assert_*()` test variant is used.  Each test check
     539increments the test step counter unless the `T_quiet_*()` test variant is used.
     540The test step counter is initialized to zero before the test case begins to
     541execute.  The `T_step_*(step, ...)` test check variants verify that the test
     542step counter is equal to the planned test step value, otherwise the test check
     543fails.
     544
     545Test Check Variant Conventions
     546~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
     547
     548The `T_quiet_*()` test check variants do not increment the test step counter
     549and only print a message if the test check fails.  This is helpful in case a
     550test check appears in a tight loop.
     551
     552The `T_step_*(step, ...)` test check variants check in addition that the test
     553step counter is equal to the specified test step value, otherwise the test
     554check fails.
     555
     556The `T_assert_*()` and `T_step_assert_*(step, ...)` test check variants stop
     557the current test case execution if the test check fails.
    510558
    511559Test Check Parameter Conventions
     
    558606passes, otherwise it fails.
    559607
    560 Test Check Variant Conventions
    561 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    562 
    563 The `T_quiet_*()` test check variants do not increment the test step counter
    564 and only print a message if the test check fails.  This is helpful in case a
    565 test check appears in a tight loop.
    566 
    567 The `T_step_*(step, ...)` test check variants check in addition that the test
    568 step counter is equal to the specified test step value, otherwise the test
    569 check fails.
    570 
    571 The `T_assert_*()` and `T_step_assert_*(step, ...)` test check variants stop
    572 the current test case execution if the test check fails.
    573 
    574 The following names for test check type variants are used:
     608Test Check Type Conventions
     609~~~~~~~~~~~~~~~~~~~~~~~~~~~
     610
     611The following names for test check types are used:
    575612
    576613ptr
     
    654691sz
    655692    The test value must be of type `size_t`.
     693
     694Integers
     695~~~~~~~~
     696
     697Let `xyz` be the type variant which shall be one of `schar`, `uchar`, `short`,
     698`ushort`, `int`, `uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`,
     699`u16`, `i32`, `u32`, `i64`, `u64`, `iptr`, `uptr`, `ssz`, and `sz`.
     700
     701Let `I` be the type name which shall be compatible to the type variant.
     702
     703The following test checks for integers are available:
     704
     705.. code-block:: c
     706
     707    void T_eq_xyz(I a, I e);
     708    void T_assert_eq_xyz(I a, I e);
     709    void T_quiet_eq_xyz(I a, I e);
     710    void T_step_eq_xyz(unsigned int step, I a, I e);
     711    void T_step_assert_eq_xyz(unsigned int step, I a, I e);
     712
     713    void T_ne_xyz(I a, I e);
     714    void T_assert_ne_xyz(I a, I e);
     715    void T_quiet_ne_xyz(I a, I e);
     716    void T_step_ne_xyz(unsigned int step, I a, I e);
     717    void T_step_assert_ne_xyz(unsigned int step, I a, I e);
     718
     719    void T_ge_xyz(I a, I e);
     720    void T_assert_ge_xyz(I a, I e);
     721    void T_quiet_ge_xyz(I a, I e);
     722    void T_step_ge_xyz(unsigned int step, I a, I e);
     723    void T_step_assert_ge_xyz(unsigned int step, I a, I e);
     724
     725    void T_gt_xyz(I a, I e);
     726    void T_assert_gt_xyz(I a, I e);
     727    void T_quiet_gt_xyz(I a, I e);
     728    void T_step_gt_xyz(unsigned int step, I a, I e);
     729    void T_step_assert_gt_xyz(unsigned int step, I a, I e);
     730
     731    void T_le_xyz(I a, I e);
     732    void T_assert_le_xyz(I a, I e);
     733    void T_quiet_le_xyz(I a, I e);
     734    void T_step_le_xyz(unsigned int step, I a, I e);
     735    void T_step_assert_le_xyz(unsigned int step, I a, I e);
     736
     737    void T_lt_xyz(I a, I e);
     738    void T_assert_lt_xyz(I a, I e);
     739    void T_quiet_lt_xyz(I a, I e);
     740    void T_step_lt_xyz(unsigned int step, I a, I e);
     741    void T_step_assert_lt_xyz(unsigned int step, I a, I e);
     742
     743An automatically generated message is printed in case the test check fails.
    656744
    657745Boolean Expressions
     
    839927An automatically generated message is printed in case the test check fails.
    840928
    841 Integers
    842 ~~~~~~~~
    843 
    844 The following test checks for integers are available:
    845 
    846 .. code-block:: c
    847 
    848     void T_eq_xyz(I a, I e);
    849     void T_assert_eq_xyz(I a, I e);
    850     void T_quiet_eq_xyz(I a, I e);
    851     void T_step_eq_xyz(unsigned int step, I a, I e);
    852     void T_step_assert_eq_xyz(unsigned int step, I a, I e);
    853 
    854     void T_ne_xyz(I a, I e);
    855     void T_assert_ne_xyz(I a, I e);
    856     void T_quiet_ne_xyz(I a, I e);
    857     void T_step_ne_xyz(unsigned int step, I a, I e);
    858     void T_step_assert_ne_xyz(unsigned int step, I a, I e);
    859 
    860     void T_ge_xyz(I a, I e);
    861     void T_assert_ge_xyz(I a, I e);
    862     void T_quiet_ge_xyz(I a, I e);
    863     void T_step_ge_xyz(unsigned int step, I a, I e);
    864     void T_step_assert_ge_xyz(unsigned int step, I a, I e);
    865 
    866     void T_gt_xyz(I a, I e);
    867     void T_assert_gt_xyz(I a, I e);
    868     void T_quiet_gt_xyz(I a, I e);
    869     void T_step_gt_xyz(unsigned int step, I a, I e);
    870     void T_step_assert_gt_xyz(unsigned int step, I a, I e);
    871 
    872     void T_le_xyz(I a, I e);
    873     void T_assert_le_xyz(I a, I e);
    874     void T_quiet_le_xyz(I a, I e);
    875     void T_step_le_xyz(unsigned int step, I a, I e);
    876     void T_step_assert_le_xyz(unsigned int step, I a, I e);
    877 
    878     void T_lt_xyz(I a, I e);
    879     void T_assert_lt_xyz(I a, I e);
    880     void T_quiet_lt_xyz(I a, I e);
    881     void T_step_lt_xyz(unsigned int step, I a, I e);
    882     void T_step_assert_lt_xyz(unsigned int step, I a, I e);
    883 
    884 The type variant `xyz` must be `schar`, `uchar`, `short`, `ushort`, `int`,
    885 `uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`, `u16`, `i32`, `u32`,
    886 `i64`, `u64`, `iptr`, `uptr`, `ssz`, or `sz`.
    887 
    888 The type name `I` must be compatible to the type variant.
    889 
    890 An automatically generated message is printed in case the test check fails.
    891 
    892929RTEMS Status Codes
    893930~~~~~~~~~~~~~~~~~~
     
    10281065forbidden in interrupt context.  The formatted output functions provided by the
    10291066test framework work in every context.
     1067
     1068Utility
     1069-------
     1070
     1071You can stop a test case via the ``T_stop()`` function.  This function does not
     1072return.  You can indicate unreachable code paths with the ``T_unreachable()``
     1073function.  If this function is called, then the test case stops.
     1074
     1075You can busy wait with the ``T_busy()`` function:
     1076
     1077.. code-block:: c
     1078
     1079    void T_busy(uint_fast32_t count);
     1080
     1081It performs a busy loop with the specified iteration count.  This function is
     1082optimized to not perform memory accesses and should have a small jitter.  The
     1083loop iterations have a processor-specific duration.
     1084
     1085You can get an iteration count for the ``T_busy()`` function which corresponds
     1086roughly to one clock tick interval with the ``T_get_one_clock_tick_busy()``
     1087function:
     1088
     1089.. code-block:: c
     1090
     1091    uint_fast32_t T_get_one_clock_tick_busy(void);
     1092
     1093This function requires a clock driver.  It must be called from thread context
     1094with interrupts enabled.  It may return a different value each time it is
     1095called.
    10301096
    10311097Time Services
     
    13541420    E:measure_empty:N:1:F:0:D:14.284869
    13551421
     1422Interrupt Tests
     1423---------------
     1424
     1425In the operating system implementation you may have two kinds of critical
     1426sections.  Firstly, there are low-level critical sections protected by
     1427interrupts disabled and maybe also some SMP spin lock.  Secondly, there are
     1428high-level critical sections which are protected by disabled thread
     1429dispatching.  The high-level critical sections may contain several low-level
     1430critical sections.  Between these low-level critical sections interrupts may
     1431happen which could alter the code path taken in the high-level critical
     1432section.
     1433
     1434The test framework provides support to write test cases for high-level critical
     1435sections though the `T_interrupt_test()` function:
     1436
     1437.. code-block:: c
     1438
     1439    typedef enum {
     1440        T_INTERRUPT_TEST_INITIAL,
     1441        T_INTERRUPT_TEST_ACTION,
     1442        T_INTERRUPT_TEST_BLOCKED,
     1443        T_INTERRUPT_TEST_CONTINUE,
     1444        T_INTERRUPT_TEST_DONE,
     1445        T_INTERRUPT_TEST_EARLY,
     1446        T_INTERRUPT_TEST_INTERRUPT,
     1447        T_INTERRUPT_TEST_LATE,
     1448        T_INTERRUPT_TEST_TIMEOUT
     1449    } T_interrupt_test_state;
     1450
     1451    typedef struct {
     1452        void                   (*prepare)(void *arg);
     1453        void                   (*action)(void *arg);
     1454        T_interrupt_test_state (*interrupt)(void *arg);
     1455        void                   (*blocked)(void *arg);
     1456        uint32_t                 max_iteration_count;
     1457    } T_interrupt_test_config;
     1458
     1459    T_interrupt_test_state T_interrupt_test(
     1460        const T_interrupt_test_config *config,
     1461        void                          *arg
     1462    );
     1463
     1464This function returns ``T_INTERRUPT_TEST_DONE`` if the test condition was
     1465satisfied within the maximum iteration count, otherwise it returns
     1466``T_INTERRUPT_TEST_TIMEOUT``.  The interrupt test run uses the specified
     1467configuration and passes the specified argument to all configured handlers.
     1468The function shall be called from thread context with interrupts enabled.
     1469
     1470.. image:: ../images/eng/interrupt-test.*
     1471    :scale: 60
     1472    :align: center
     1473
     1474The interrupt test uses an *adaptive bisection algorithm* to try to hit the
     1475code section under test by an interrupt.  In each test iteration, it waits for
     1476a time point one quarter of the clock tick interval after a clock tick using
     1477the monotonic clock.  Then it performs a busy wait using ``T_busy()`` with a
     1478busy count controlled by the adaptive bisection algorithm.  The test maintains
     1479a sample set of upper and lower bound busy wait count values.  Initially, the
     1480lower bound values are zero and the upper bound values are set to a value
     1481returned by ``T_get_one_clock_tick_busy()``.  The busy wait count for an
     1482iteration is set to the middle point between the arithmetic mean of the lower
     1483and upper bound sample values.  After the action handler returns, the set of
     1484lower and upper bound sample values is updated based on the test state.  If the
     1485test state is ``T_INTERRUPT_TEST_EARLY``, then the oldest upper bound sample
     1486value is replaced by the busy wait count used to delay the action and the
     1487latest lower bound sample value is slightly decreased.  Reducing the lower
     1488bound helps to avoid a zero length interval between the upper and lower bounds.
     1489If the test state is ``T_INTERRUPT_TEST_LATE``, then the oldest lower bound
     1490sample value is replaced by the busy wait count used to delay the action and
     1491the latest upper bound sample value is slightly increased.  In all other test
     1492states the timing values remain as is.  Using the arithmetic mean of a sample
     1493set dampens the effect of each test iteration and is an heuristic to mitigate
     1494the influence of jitters in the action code execution.
     1495
     1496The optional *prepare* handler should prepare the system so that the *action*
     1497handler can be called.  It is called in a tight loop, so all the time consuming
     1498setup should be done before ``T_interrupt_test()`` is called.  During the
     1499preparation the test state is ``T_INTERRUPT_TEST_INITIAL``.  The preparation
     1500handler shall not change the test state.
     1501
     1502The *action* handler should call the function which executes the code section
     1503under test.  The execution path up to the code section under test should have a
     1504low jitter.  Otherwise, the adaptive bisection algorithm may not find the right
     1505spot.
     1506
     1507The *interrupt* handler should check if the test condition is satisfied or a
     1508new iteration is necessary.  This handler is called in interrupt context.  It
     1509shall return ``T_INTERRUPT_TEST_DONE`` if the test condition is satisfied and
     1510the test run is done.  It shall return ``T_INTERRUPT_TEST_EARLY`` if the
     1511interrupt happened too early to satisfy the test condition.  It shall return
     1512``T_INTERRUPT_TEST_LATE`` if the interrupt happened too late to satisfy the
     1513test condition.  It shall return ``T_INTERRUPT_TEST_CONTINUE`` if the test
     1514should continue with the current timing settings.  Other states shall not be
     1515returned.  It is critical to return the early and late states if the test
     1516condition was not satisfied, otherwise the adaptive bisection algorithm may not
     1517work.  The returned state is used to try to change the test state from
     1518``T_INTERRUPT_TEST_ACTION`` to the returned state.
     1519
     1520The optional *blocked* handler is invoked if the executing thread blocks during
     1521the action processing.  It should remove the blocking condition of the thread
     1522so that the next iteration can start.  It can use
     1523``T_interrupt_change_state()`` to change the interrupt test state.
     1524
     1525The *max iteration count* configuration member defines the maximum iteration
     1526count of the test loop.  If the maximum iteration count is reached before the
     1527test condition is satisfied, then ``T_interrupt_test()`` returns
     1528``T_INTERRUPT_TEST_TIMEOUT``.
     1529
     1530The *interrupt* and *blocked* handlers may be called in arbitrary test states.
     1531
     1532The *action*, *interrupt*, and *blocked* handlers can use
     1533``T_interrupt_test_get_state()`` to get the current test state:
     1534
     1535.. code-block:: c
     1536
     1537    T_interrupt_test_state T_interrupt_test_get_state(void);
     1538
     1539The *action*, *interrupt*, and *blocked* handlers can use
     1540``T_interrupt_test_change_state()`` to try to change the test state from an
     1541expected state to a desired state:
     1542
     1543.. code-block:: c
     1544
     1545    T_interrupt_test_state T_interrupt_test_change_state(
     1546        T_interrupt_test_state expected_state,
     1547        T_interrupt_test_state desired_state
     1548    );
     1549
     1550The function returns the previous state.  If it **differs from the expected
     1551state**, then the requested state **change to the desired state did not take
     1552place**.  In an SMP configuration, do not call this function in a tight loop.
     1553It could lock up the test run.  To busy wait for a state change, use
     1554``T_interrupt_test_get_state()``.
     1555
     1556The *action* handler can use ``T_interrupt_test_busy_wait_for_interrupt()`` to
     1557busy wait for the interrupt:
     1558
     1559.. code-block:: c
     1560
     1561    void T_interrupt_test_busy_wait_for_interrupt(void);
     1562
     1563This is useful if the action code does not block to wait for the interrupt.  If
     1564the action handler just returns the test code immediately prepares the next
     1565iteration and may miss an interrupt which happens too late.
    13561566
    13571567Test Runner
Note: See TracChangeset for help on using the changeset viewer.