1 | .. comment SPDX-License-Identifier: CC-BY-SA-4.0 |
---|
2 | |
---|
3 | .. Copyright (C) 2018 Chris Johns <chrisj@rtems.org> |
---|
4 | |
---|
5 | Test Banners |
---|
6 | ------------ |
---|
7 | |
---|
8 | All test output banners or strings are embedded in each test and the test |
---|
9 | outputs them to the BSP's console as it executes. The RTEMS Tester captures the |
---|
10 | BSP's console and uses this information to manage the state of the executing |
---|
11 | test. The banner strings are: |
---|
12 | |
---|
13 | .. _test-banner-begin: |
---|
14 | .. index:: test begin, TEST BEGIN |
---|
15 | |
---|
16 | ``*** BEGIN TEST <name> ***`` |
---|
17 | The test has loaded, RTEMS has initialized and the test specific code is |
---|
18 | about to start executing. The ``<name>`` field is the name of the test. The |
---|
19 | test name is internal to the test and may not match the name of the |
---|
20 | executable. The test name is informative and not used by the RTEMS Tester. |
---|
21 | |
---|
22 | .. _test-banner-end: |
---|
23 | .. index:: test end, TEST END |
---|
24 | |
---|
25 | ``*** END TEST <name> ***`` |
---|
26 | The test has finished without error and has passed. The ``<name>`` field is |
---|
27 | the name of the test. See the :ref:`Test Begin Banner <test-banner-begin>` |
---|
28 | for details about the name. |
---|
29 | |
---|
30 | .. index:: test banner version, TEST VERSION |
---|
31 | |
---|
32 | ``*** TEST VERSION: <version>`` |
---|
33 | The test prints the RTEMS version return by the RTEMS Version API as |
---|
34 | ``<version>``. All tests must match the first test's version or the Wrong |
---|
35 | Version error count is incremented. |
---|
36 | |
---|
37 | .. _test-banner-state: |
---|
38 | .. index:: test state, TEST STATE |
---|
39 | |
---|
40 | ``*** TEST STATE: <state>`` |
---|
41 | The test is tagged in the RTEMS sources with a special ``<state>`` for this |
---|
42 | BSP. See :ref:`Test States <test-states>` for the list of possible |
---|
43 | states. The state banner lets the RTEMS Tester categorize and manage the |
---|
44 | test. For example a user input test typically needing user interaction may |
---|
45 | never complete producing an *invalid* test result. A user input test is |
---|
46 | terminated to avoid extended delays in a long test run. |
---|
47 | |
---|
48 | .. _test-banner-build: |
---|
49 | .. index:: test build, TEST BUILD |
---|
50 | |
---|
51 | ``*** TEST BUILD: <build>`` |
---|
52 | The test prints the RTEMS build as a space separated series of labels as |
---|
53 | ``<build>``. The build labels are created from the configuration settings in |
---|
54 | the Super Score header file ``rtems/score/cputops.h``. All tests must match |
---|
55 | the first test's build or the Wrong Build error count is incremented. |
---|
56 | |
---|
57 | .. _test-banner-tools: |
---|
58 | .. index:: test tools, TEST TOOLS |
---|
59 | |
---|
60 | ``*** TEST TOOLS: <version>`` |
---|
61 | The test prints the RTEMS tools version returned the GGC internal macro |
---|
62 | ``_VERSION_`` as ``<version>``. All tests must match the first test's tools |
---|
63 | version string or the Wrong Tools error count is incremented. |
---|
64 | |
---|
65 | .. _test-states: |
---|
66 | .. index:: Test states |
---|
67 | |
---|
68 | Test States |
---|
69 | ----------- |
---|
70 | |
---|
71 | The tests states are: |
---|
72 | |
---|
73 | .. index:: test state passed |
---|
74 | ``passed`` |
---|
75 | The test start and end banners have been sent to the console. |
---|
76 | |
---|
77 | .. index:: test state failure |
---|
78 | ``failure`` |
---|
79 | The test start banner has been sent to the console and no end banner has been |
---|
80 | seen when a target restart is detected. |
---|
81 | |
---|
82 | .. index:: test state expected-fail |
---|
83 | ``excepted-fail`` |
---|
84 | The test is tagged as ``expected-fail`` in the RTEMS sources for this BSP and |
---|
85 | outputs the banner ``*** TEST STATE: EXPECTED_FAIL``. The test is known not |
---|
86 | to pass on this BSP. The RTEMS Tester will let the test run as far as it |
---|
87 | can and if the test passes it is recorded as a pass in the test results |
---|
88 | otherwise it is recorded as *expected-fail*. |
---|
89 | |
---|
90 | .. index:: test state indeterminate |
---|
91 | ``indeterminate`` |
---|
92 | The test is tagged as ``indeterminate`` in the RTEMS sources for this BSP and |
---|
93 | outputs the banner ``*** TEST STATE: INDETERMINATE``. The test may or may not |
---|
94 | pass so the result is not able to be determined. The RTEMS Tester will let |
---|
95 | the test run as far as it can and record the result as indeterminate. |
---|
96 | |
---|
97 | .. index:: test state user-input |
---|
98 | ``user-input`` |
---|
99 | The test is tagged as ``user-input`` in the RTEMS sources and outputs the |
---|
100 | banner ``*** TEST STATE: USER_INPUT``. The RTEMS Tester will reset the target |
---|
101 | if the target's configuration provides a target reset command. |
---|
102 | |
---|
103 | .. index:: test state benchmark |
---|
104 | ``benchmark`` |
---|
105 | The test is tagged as ``benchmark`` in the RTEMS sources and outputs the |
---|
106 | banner ``*** TEST STATE: BENCHMARK``. Benchmarks can take a while to run and |
---|
107 | performance is not regression tested in RTEMS. The RTEMS Tester will reset |
---|
108 | the target if the target's configuration provides a target reset command. |
---|
109 | |
---|
110 | .. index:: test state timeout |
---|
111 | ``timeout`` |
---|
112 | The test start banner has been sent to the console and no end banner is seen |
---|
113 | within the *timeout* period and the target has not restart. A default |
---|
114 | *timeout* can be set in a target configuration, a user configuration or |
---|
115 | provide on the RTEMS Tester's command line using the ``--timeout`` option. |
---|
116 | |
---|
117 | .. index:: test state invalid |
---|
118 | ``invalid`` |
---|
119 | The test did not output a start banner and the RTEMS Tester has detected the |
---|
120 | target has restarted. This means the executable did not load correctly, the |
---|
121 | RTEMS kernel did not initialize or the RTEMS kernel configuration failed for |
---|
122 | this BSP. |
---|
123 | |
---|
124 | Expected Test States |
---|
125 | ^^^^^^^^^^^^^^^^^^^^ |
---|
126 | |
---|
127 | A test's expected state is set in the RTEMS kernel's testsuite. The default for |
---|
128 | a tested is to ``pass``. If a test is known to fail it can have it's state set |
---|
129 | to ``expected-fail``. Setting tests that are known to fail to ``expected-fail`` |
---|
130 | lets everyone know a failure is not to be countered and consider a regression. |
---|
131 | |
---|
132 | Expected test states are list in test configuration files that end with the |
---|
133 | file extension ``.tcfg``. The testsuite supports global test configurations in |
---|
134 | the ``testsuite/testdata`` directory. Global test states are applied to all |
---|
135 | BSPs. BSPs can provide a test configuration that applies to just that BSP. |
---|
136 | |
---|
137 | The test configuration file format is:: |
---|
138 | |
---|
139 | state: test test test |
---|
140 | |
---|
141 | where ``test test test`` is a list of tests the state applies too. The ``state`` is one |
---|
142 | of: |
---|
143 | |
---|
144 | ``include`` |
---|
145 | The test list is the name of a test configuration file to include |
---|
146 | |
---|
147 | ``exclude`` |
---|
148 | The tests listed are not build. This can happen if a BSP cannot support a |
---|
149 | test. For example it does not have enough memory. |
---|
150 | |
---|
151 | ``expected-fail`` |
---|
152 | The tests listed are set to expected fail. The test will fail on the BSP |
---|
153 | being built. |
---|
154 | |
---|
155 | ``user-input`` |
---|
156 | The tests listed require user input to run and are not supported by automatic |
---|
157 | testers. |
---|
158 | |
---|
159 | ``indeterminate`` |
---|
160 | The tests listed may pass or may not, the result is not reliable. |
---|
161 | |
---|
162 | ``benchmark`` |
---|
163 | The tests listed are benchmarks. Benchmarks are flagged and not left to |
---|
164 | run to completion because they may take too long. |
---|
165 | |
---|
166 | |
---|
167 | |
---|
168 | Test Builds |
---|
169 | ----------- |
---|
170 | |
---|
171 | The test reports the build of RTEMS being tested. The build are: |
---|
172 | |
---|
173 | .. index:: build default |
---|
174 | ``default`` |
---|
175 | The build is the default. No RTEMS configure options have been used. |
---|
176 | |
---|
177 | .. index:: build posix |
---|
178 | ``posix`` |
---|
179 | The build includes the POSIX API. The RTEMS configure option |
---|
180 | ``--enable-posix`` has been used. The ``cpuopts.h`` define ``RTEMS_POSIX`` |
---|
181 | has defined and it true. |
---|
182 | |
---|
183 | .. index:: build smp |
---|
184 | ``smp`` |
---|
185 | The build is an SMP kernel. The RTEMS configure option ``--enable-smp`` has |
---|
186 | been used. The ``cpuopts.h`` define ``RTEMS_SMP`` has defined and it true. |
---|
187 | |
---|
188 | .. index:: build mp |
---|
189 | ``mp`` |
---|
190 | The build is an MP kernel. The RTEMS configure option |
---|
191 | ``--enable-multiprocessing`` has been used. The ``cpuopts.h`` define |
---|
192 | ``RTEMS_MULTIPROCESSING`` has defined and it true. |
---|
193 | |
---|
194 | .. index:: build paravirt |
---|
195 | ``paravirt`` |
---|
196 | The build is a paravirtualization kernel. The ``cpuopts.h`` define |
---|
197 | ``RTEMS_PARAVIRT`` has defined and it true. |
---|
198 | |
---|
199 | .. index:: build debug |
---|
200 | ``debug`` |
---|
201 | The build includes kernel debugging support. The RTEMS configure option |
---|
202 | ``--enable-debug`` has been used. The ``cpuopts.h`` define ``RTEMS_DEBUG`` |
---|
203 | has defined and it true. |
---|
204 | |
---|
205 | .. index:: build profiling |
---|
206 | ``profiling`` |
---|
207 | The build include profiling support. The RTEMS configure option |
---|
208 | ``--enable-profiling`` has been used. The ``cpuopts.h`` define |
---|
209 | ``RTEMS_PROFILING`` has defined and it true. |
---|