1 | .. SPDX-License-Identifier: CC-BY-SA-4.0 |
---|
2 | |
---|
3 | .. Copyright (C) 2018, 2019 embedded brains GmbH |
---|
4 | .. Copyright (C) 2018, 2019 Sebastian Huber |
---|
5 | |
---|
6 | Software Test Framework |
---|
7 | *********************** |
---|
8 | |
---|
9 | .. _RTEMSTestFramework: |
---|
10 | |
---|
11 | The RTEMS Test Framework |
---|
12 | ======================== |
---|
13 | |
---|
14 | The `RTEMS Test Framework` helps you to write test suites. It has the following |
---|
15 | features: |
---|
16 | |
---|
17 | * Implemented in standard C11 |
---|
18 | |
---|
19 | * Runs on at least FreeBSD, MSYS2, Linux and RTEMS |
---|
20 | |
---|
21 | * Test runner and test case code can be in separate translation units |
---|
22 | |
---|
23 | * Test cases are automatically registered at link-time |
---|
24 | |
---|
25 | * Test cases may have a test fixture |
---|
26 | |
---|
27 | * Test checks for various standard types |
---|
28 | |
---|
29 | * Supports test case planning |
---|
30 | |
---|
31 | * Test case scoped dynamic memory |
---|
32 | |
---|
33 | * Test case destructors |
---|
34 | |
---|
35 | * Test case resource accounting to show that no resources are leaked |
---|
36 | during the test case execution |
---|
37 | |
---|
38 | * Supports early test case exit, e.g. in case a malloc() fails |
---|
39 | |
---|
40 | * Individual test case and overall test suite duration is reported |
---|
41 | |
---|
42 | * Procedures for code runtime measurements in RTEMS |
---|
43 | |
---|
44 | * Easy to parse test report to generate for example human readable test reports |
---|
45 | |
---|
46 | * Low overhead time measurement of short time sequences (using cycle counter |
---|
47 | hardware if a available) |
---|
48 | |
---|
49 | * Configurable time service provider for a monotonic clock |
---|
50 | |
---|
51 | * Low global memory overhead for test cases and test checks |
---|
52 | |
---|
53 | * Supports multi-threaded execution and interrupts in test cases |
---|
54 | |
---|
55 | * A simple (polled) put character function is sufficient to produce the test report |
---|
56 | |
---|
57 | * Only text, global data and a stack pointer must be set up to run a test suite |
---|
58 | |
---|
59 | * No dynamic memory is used by the framework itself |
---|
60 | |
---|
61 | * No memory is aggregated throughout the test case execution |
---|
62 | |
---|
63 | Nomenclature |
---|
64 | ------------ |
---|
65 | |
---|
66 | A `test suite` is a collection of test cases. A `test case` consists of |
---|
67 | individual test actions and checks. A `test check` determines if the outcome |
---|
68 | of a test action meets its expectation. A `test action` is a program sequence |
---|
69 | with an observable outcome, for example a function invocation with a return |
---|
70 | status. If the test action outcome is all right, then the test check passes, |
---|
71 | otherwise the test check fails. The test check failures of a test case are |
---|
72 | summed up. A test case passes, if the failure count of this test case is zero, |
---|
73 | otherwise the test case fails. The test suite passes if all test cases pass, |
---|
74 | otherwise it fails. |
---|
75 | |
---|
76 | Test Cases |
---|
77 | ---------- |
---|
78 | |
---|
79 | You can write a test case with the `T_TEST_CASE()` macro followed by a function |
---|
80 | body: |
---|
81 | |
---|
82 | .. code-block:: c |
---|
83 | |
---|
84 | T_TEST_CASE(name) |
---|
85 | { |
---|
86 | /* Your test case code */ |
---|
87 | } |
---|
88 | |
---|
89 | The test case `name` must be a valid C designator. The test case names must be |
---|
90 | unique within the test suite. Just link modules with test cases to the test |
---|
91 | runner to form a test suite. The test cases are automatically registered via |
---|
92 | static constructors. |
---|
93 | |
---|
94 | .. code-block:: c |
---|
95 | :caption: Test Case Example |
---|
96 | |
---|
97 | #include <t.h> |
---|
98 | |
---|
99 | static int add(int a, int b) |
---|
100 | { |
---|
101 | return a + b; |
---|
102 | } |
---|
103 | |
---|
104 | T_TEST_CASE(a_test_case) |
---|
105 | { |
---|
106 | int actual_value; |
---|
107 | |
---|
108 | actual_value = add(1, 1); |
---|
109 | T_eq_int(actual_value, 2); |
---|
110 | T_true(false, "a test failure message"); |
---|
111 | } |
---|
112 | |
---|
113 | .. code-block:: none |
---|
114 | :caption: Test Case Report |
---|
115 | |
---|
116 | B:a_test_case |
---|
117 | P:0:8:UI1:test-simple.c:13 |
---|
118 | F:1:8:UI1:test-simple.c:14:a test failure message |
---|
119 | E:a_test_case:N:2:F:1:D:0.001657 |
---|
120 | |
---|
121 | The `B` line indicates the begin of test case `a_test_case`. The `P` line |
---|
122 | shows that the test check in file `test-simple.c` at line 13 executed by task |
---|
123 | `UI1` on processor 0 as the test step 0 passed. The invocation of `add()` in |
---|
124 | line 12 is the test action of test step 0. The `F` lines shows that the test |
---|
125 | check in file `test-simple.c` at line 14 executed by task `UI1` on processor 0 |
---|
126 | as the test step 1 failed with a message of `"a test failure message"`. The |
---|
127 | `E` line indicates the end of test case `a_test_case` resulting in a total of |
---|
128 | two test steps (`N`) and one test failure (`F`). The test case execution |
---|
129 | duration (`D`) was 0.001657 seconds. For test report details see: |
---|
130 | :ref:`Test Reporting <RTEMSTestFrameworkTestReporting>`. |
---|
131 | |
---|
132 | Test Fixture |
---|
133 | ------------ |
---|
134 | |
---|
135 | You can write a test case with a test fixture with the `T_TEST_CASE_FIXTURE()` |
---|
136 | macro followed by a function body: |
---|
137 | |
---|
138 | .. code-block:: c |
---|
139 | |
---|
140 | T_TEST_CASE_FIXTURE(name, fixture) |
---|
141 | { |
---|
142 | /* Your test case code */ |
---|
143 | } |
---|
144 | |
---|
145 | The test case `name` must be a valid C designator. The test case names must be |
---|
146 | unique within the test suite. The `fixture` must point to a statically |
---|
147 | initialized read-only object of type `T_fixture`. The test fixture |
---|
148 | provides methods to setup, stop and tear down a test case. A context is passed |
---|
149 | to the methods. The initial context is defined by the read-only fixture |
---|
150 | object. The context can be obtained by the `T_fixture_context()` |
---|
151 | function. It can be set within the scope of one test case by the |
---|
152 | `T_set_fixture_context()` function. This can be used for example to |
---|
153 | dynamically allocate a test environment in the setup method. |
---|
154 | |
---|
155 | .. code-block:: c |
---|
156 | :caption: Test Fixture Example |
---|
157 | |
---|
158 | #include <t.h> |
---|
159 | |
---|
160 | static int initial_value = 3; |
---|
161 | |
---|
162 | static int counter; |
---|
163 | |
---|
164 | static void |
---|
165 | setup(void *ctx) |
---|
166 | { |
---|
167 | int *c; |
---|
168 | |
---|
169 | T_log(T_QUIET, "setup begin"); |
---|
170 | T_eq_ptr(ctx, &initial_value); |
---|
171 | T_eq_ptr(ctx, T_fixture_context()); |
---|
172 | c = ctx; |
---|
173 | counter = *c; |
---|
174 | T_set_fixture_context(&counter); |
---|
175 | T_eq_ptr(&counter, T_fixture_context()); |
---|
176 | T_log(T_QUIET, "setup end"); |
---|
177 | } |
---|
178 | |
---|
179 | static void |
---|
180 | stop(void *ctx) |
---|
181 | { |
---|
182 | int *c; |
---|
183 | |
---|
184 | T_log(T_QUIET, "stop begin"); |
---|
185 | T_eq_ptr(ctx, &counter); |
---|
186 | c = ctx; |
---|
187 | ++(*c); |
---|
188 | T_log(T_QUIET, "stop end"); |
---|
189 | } |
---|
190 | |
---|
191 | static void |
---|
192 | teardown(void *ctx) |
---|
193 | { |
---|
194 | int *c; |
---|
195 | |
---|
196 | T_log(T_QUIET, "teardown begin"); |
---|
197 | T_eq_ptr(ctx, &counter); |
---|
198 | c = ctx; |
---|
199 | T_eq_int(*c, 4); |
---|
200 | T_log(T_QUIET, "teardown end"); |
---|
201 | } |
---|
202 | |
---|
203 | static const T_fixture fixture = { |
---|
204 | .setup = setup, |
---|
205 | .stop = stop, |
---|
206 | .teardown = teardown, |
---|
207 | .initial_context = &initial_value |
---|
208 | }; |
---|
209 | |
---|
210 | T_TEST_CASE_FIXTURE(fixture, &fixture) |
---|
211 | { |
---|
212 | T_assert_true(true, "all right"); |
---|
213 | T_assert_true(false, "test fails and we stop the test case"); |
---|
214 | T_log(T_QUIET, "not reached"); |
---|
215 | } |
---|
216 | |
---|
217 | .. code-block:: none |
---|
218 | :caption: Test Fixture Report |
---|
219 | |
---|
220 | B:fixture |
---|
221 | L:setup begin |
---|
222 | P:0:0:UI1:test-fixture.c:13 |
---|
223 | P:1:0:UI1:test-fixture.c:14 |
---|
224 | P:2:0:UI1:test-fixture.c:18 |
---|
225 | L:setup end |
---|
226 | P:3:0:UI1:test-fixture.c:55 |
---|
227 | F:4:0:UI1:test-fixture.c:56:test fails and we stop the test case |
---|
228 | L:stop begin |
---|
229 | P:5:0:UI1:test-fixture.c:28 |
---|
230 | L:stop end |
---|
231 | L:teardown begin |
---|
232 | P:6:0:UI1:test-fixture.c:40 |
---|
233 | P:7:0:UI1:test-fixture.c:42 |
---|
234 | L:teardown end |
---|
235 | E:fixture:N:8:F:1 |
---|
236 | |
---|
237 | Test Case Planning |
---|
238 | ------------------ |
---|
239 | |
---|
240 | Each non-quiet test check fetches and increments the test step counter |
---|
241 | atomically. For each test case execution the planned steps can be specified |
---|
242 | with the `T_plan()` function. |
---|
243 | |
---|
244 | .. code-block:: c |
---|
245 | |
---|
246 | void T_plan(unsigned int planned_steps); |
---|
247 | |
---|
248 | This function must be invoked at most once in each test case execution. If the |
---|
249 | planned test steps are set with this function, then the final test steps after |
---|
250 | the test case execution must be equal to the planned steps, otherwise the test |
---|
251 | case fails. |
---|
252 | |
---|
253 | Use the `T_step_*(step, ...)` test check variants to ensure that the test case |
---|
254 | execution follows exactly the planned steps. |
---|
255 | |
---|
256 | .. code-block:: c |
---|
257 | :caption: Test Planning Example |
---|
258 | |
---|
259 | #include <t.h> |
---|
260 | |
---|
261 | T_TEST_CASE(wrong_step) |
---|
262 | { |
---|
263 | T_plan(2); |
---|
264 | T_step_true(0, true, "all right"); |
---|
265 | T_step_true(2, true, "wrong step"); |
---|
266 | } |
---|
267 | |
---|
268 | T_TEST_CASE(plan_ok) |
---|
269 | { |
---|
270 | T_plan(1); |
---|
271 | T_step_true(0, true, "all right"); |
---|
272 | } |
---|
273 | |
---|
274 | T_TEST_CASE(plan_failed) |
---|
275 | { |
---|
276 | T_plan(2); |
---|
277 | T_step_true(0, true, "not enough steps"); |
---|
278 | T_quiet_true(true, "quiet test do not count"); |
---|
279 | } |
---|
280 | |
---|
281 | T_TEST_CASE(double_plan) |
---|
282 | { |
---|
283 | T_plan(99); |
---|
284 | T_plan(2); |
---|
285 | } |
---|
286 | |
---|
287 | T_TEST_CASE(steps) |
---|
288 | { |
---|
289 | T_step(0, "a"); |
---|
290 | T_plan(3); |
---|
291 | T_step(1, "b"); |
---|
292 | T_step(2, "c"); |
---|
293 | } |
---|
294 | |
---|
295 | .. code-block:: none |
---|
296 | :caption: Test Planning Report |
---|
297 | |
---|
298 | B:wrong_step |
---|
299 | P:0:0:UI1:test-plan.c:6 |
---|
300 | F:1:0:UI1:test-plan.c:7:planned step (2) |
---|
301 | E:wrong_step:N:2:F:1 |
---|
302 | B:plan_ok |
---|
303 | P:0:0:UI1:test-plan.c:13 |
---|
304 | E:plan_ok:N:1:F:0 |
---|
305 | B:plan_failed |
---|
306 | P:0:0:UI1:test-plan.c:19 |
---|
307 | F:*:0:UI1:*:*:actual steps (1), planned steps (2) |
---|
308 | E:plan_failed:N:1:F:1 |
---|
309 | B:double_plan |
---|
310 | F:*:0:UI1:*:*:planned steps (99) already set |
---|
311 | E:double_plan:N:0:F:1 |
---|
312 | B:steps |
---|
313 | P:0:0:UI1:test-plan.c:31 |
---|
314 | P:1:0:UI1:test-plan.c:33 |
---|
315 | P:2:0:UI1:test-plan.c:34 |
---|
316 | E:steps:N:3:F:0 |
---|
317 | |
---|
318 | Test Case Resource Accounting |
---|
319 | ----------------------------- |
---|
320 | |
---|
321 | The framework can check if various resources are leaked during a test case |
---|
322 | execution. The resource checkers are specified by the test run configuration. |
---|
323 | On RTEMS, checks for the following resources are available |
---|
324 | |
---|
325 | * workspace and heap memory, |
---|
326 | * file descriptors, |
---|
327 | * POSIX keys and key value pairs, |
---|
328 | * RTEMS barriers, |
---|
329 | * RTEMS user extensions, |
---|
330 | * RTEMS message queues, |
---|
331 | * RTEMS partitions, |
---|
332 | * RTEMS periods, |
---|
333 | * RTEMS regions, |
---|
334 | * RTEMS semaphores, |
---|
335 | * RTEMS tasks, and |
---|
336 | * RTEMS timers. |
---|
337 | |
---|
338 | .. code-block:: c |
---|
339 | :caption: Resource Accounting Example |
---|
340 | |
---|
341 | #include <t.h> |
---|
342 | |
---|
343 | #include <stdlib.h> |
---|
344 | |
---|
345 | #include <rtems.h> |
---|
346 | |
---|
347 | T_TEST_CASE(missing_sema_delete) |
---|
348 | { |
---|
349 | rtems_status_code sc; |
---|
350 | rtems_id id; |
---|
351 | |
---|
352 | sc = rtems_semaphore_create(rtems_build_name('S', 'E', 'M', 'A'), 0, |
---|
353 | RTEMS_COUNTING_SEMAPHORE, 0, &id); |
---|
354 | T_rsc_success(sc); |
---|
355 | } |
---|
356 | |
---|
357 | T_TEST_CASE(missing_free) |
---|
358 | { |
---|
359 | void *p; |
---|
360 | |
---|
361 | p = malloc(1); |
---|
362 | T_not_null(p); |
---|
363 | } |
---|
364 | |
---|
365 | .. code-block:: none |
---|
366 | :caption: Resource Accounting Report |
---|
367 | |
---|
368 | B:missing_sema_delete |
---|
369 | P:0:0:UI1:test-leak.c:14 |
---|
370 | F:*:0:UI1:*:*:RTEMS semaphore leak (1) |
---|
371 | E:missing_sema_delete:N:1:F:1:D:0.004013 |
---|
372 | B:missing_free |
---|
373 | P:0:0:UI1:test-leak.c:22 |
---|
374 | F:*:0:UI1:*:*:memory leak in workspace or heap |
---|
375 | E:missing_free:N:1:F:1:D:0.003944 |
---|
376 | |
---|
377 | Test Case Scoped Dynamic Memory |
---|
378 | ------------------------------- |
---|
379 | |
---|
380 | You can allocate dynamic memory which is automatically freed after the current |
---|
381 | test case execution. You can provide an optional destroy function to |
---|
382 | `T_zalloc()` which is called right before the memory is freed. The |
---|
383 | `T_zalloc()` function initializes the memory to zero. |
---|
384 | |
---|
385 | .. code-block:: c |
---|
386 | |
---|
387 | void *T_malloc(size_t size); |
---|
388 | |
---|
389 | void *T_calloc(size_t nelem, size_t elsize); |
---|
390 | |
---|
391 | void *T_zalloc(size_t size, void (*destroy)(void *)); |
---|
392 | |
---|
393 | void T_free(void *ptr); |
---|
394 | |
---|
395 | .. code-block:: c |
---|
396 | :caption: Test Case Scoped Dynamic Memory Example |
---|
397 | |
---|
398 | #include <t.h> |
---|
399 | |
---|
400 | T_TEST_CASE(malloc_free) |
---|
401 | { |
---|
402 | void *p; |
---|
403 | |
---|
404 | p = T_malloc(1); |
---|
405 | T_assert_not_null(p); |
---|
406 | T_free(p); |
---|
407 | } |
---|
408 | |
---|
409 | T_TEST_CASE(malloc_auto) |
---|
410 | { |
---|
411 | void *p; |
---|
412 | |
---|
413 | p = T_malloc(1); |
---|
414 | T_assert_not_null(p); |
---|
415 | } |
---|
416 | |
---|
417 | static void |
---|
418 | destroy(void *p) |
---|
419 | { |
---|
420 | int *i; |
---|
421 | |
---|
422 | i = p; |
---|
423 | T_step_eq_int(2, *i, 1); |
---|
424 | } |
---|
425 | |
---|
426 | T_TEST_CASE(zalloc_auto) |
---|
427 | { |
---|
428 | int *i; |
---|
429 | |
---|
430 | T_plan(3); |
---|
431 | i = T_zalloc(sizeof(*i), destroy); |
---|
432 | T_step_assert_not_null(0, i); |
---|
433 | T_step_eq_int(1, *i, 0); |
---|
434 | *i = 1; |
---|
435 | } |
---|
436 | |
---|
437 | .. code-block:: none |
---|
438 | :caption: Test Case Scoped Dynamic Memory Report |
---|
439 | |
---|
440 | B:malloc_free |
---|
441 | P:0:0:UI1:test-malloc.c:8 |
---|
442 | E:malloc_free:N:1:F:0:D:0.005200 |
---|
443 | B:malloc_auto |
---|
444 | P:0:0:UI1:test-malloc.c:17 |
---|
445 | E:malloc_auto:N:1:F:0:D:0.004790 |
---|
446 | B:zalloc_auto |
---|
447 | P:0:0:UI1:test-malloc.c:35 |
---|
448 | P:1:0:UI1:test-malloc.c:36 |
---|
449 | P:2:0:UI1:test-malloc.c:26 |
---|
450 | E:zalloc_auto:N:3:F:0:D:0.006583 |
---|
451 | |
---|
452 | Test Case Destructors |
---|
453 | --------------------- |
---|
454 | |
---|
455 | You can add test case destructors with `T_add_destructor()`. They are called |
---|
456 | automatically at the test case end before the resource accounting takes place. |
---|
457 | Optionally, a registered destructor can be removed before the test case end |
---|
458 | with `T_remove_destructor()`. The `T_destructor` structure of a destructor |
---|
459 | must exist after the return from the test case body. Do not use stack memory |
---|
460 | or dynamic memory obtained via `T_malloc()`, `T_calloc()` or `T_zalloc()` for |
---|
461 | the `T_destructor` structure. |
---|
462 | |
---|
463 | .. code-block:: c |
---|
464 | |
---|
465 | void T_add_destructor(T_destructor *destructor, |
---|
466 | void (*destroy)(T_destructor *)); |
---|
467 | |
---|
468 | void T_remove_destructor(T_destructor *destructor); |
---|
469 | |
---|
470 | .. code-block:: c |
---|
471 | :caption: Test Case Destructor Example |
---|
472 | |
---|
473 | #include <t.h> |
---|
474 | |
---|
475 | static void |
---|
476 | destroy(T_destructor *dtor) |
---|
477 | { |
---|
478 | (void)dtor; |
---|
479 | T_step(0, "destroy"); |
---|
480 | } |
---|
481 | |
---|
482 | T_TEST_CASE(destructor) |
---|
483 | { |
---|
484 | static T_destructor dtor; |
---|
485 | |
---|
486 | T_plan(1); |
---|
487 | T_add_destructor(&dtor, destroy); |
---|
488 | } |
---|
489 | |
---|
490 | .. code-block:: none |
---|
491 | :caption: Test Case Destructor Report |
---|
492 | |
---|
493 | B:destructor |
---|
494 | P:0:0:UI1:test-destructor.c:7 |
---|
495 | E:destructor:N:1:F:0:D:0.003714 |
---|
496 | |
---|
497 | Test Checks |
---|
498 | ----------- |
---|
499 | |
---|
500 | A `test check` determines if the actual value presented to the test check meets |
---|
501 | its expectation. The actual value should represent the outcome of a test |
---|
502 | action. If the actual value is all right, then the test check passes, |
---|
503 | otherwise the test check fails. A failed test check does not stop the test |
---|
504 | case execution immediately unless the `T_assert_*()` test variant is used. |
---|
505 | Each test check increments the test step counter unless the `T_quiet_*()` test |
---|
506 | variant is used. The test step counter is initialized to zero before the test |
---|
507 | case begins to execute. The `T_step_*(step, ...)` test check variants verify |
---|
508 | that the test step counter is equal to the planned test step value, otherwise |
---|
509 | the test check fails. |
---|
510 | |
---|
511 | Test Check Parameter Conventions |
---|
512 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
---|
513 | |
---|
514 | The following names for test check parameters are used throughout the test |
---|
515 | checks: |
---|
516 | |
---|
517 | step |
---|
518 | The planned test step for this test check. |
---|
519 | |
---|
520 | a |
---|
521 | The actual value to check against an expected value. It is usually the |
---|
522 | first parameter in all test checks, except in the `T_step_*(step, ...)` |
---|
523 | test check variants, here it is the second parameter. |
---|
524 | |
---|
525 | e |
---|
526 | The expected value of a test check. This parameter is optional. Some test |
---|
527 | checks have an implicit expected value. If present, then this parameter is |
---|
528 | directly after the actual value parameter of the test check. |
---|
529 | |
---|
530 | fmt |
---|
531 | A printf()-like format string. Floating-point and exotic formats may be |
---|
532 | not supported. |
---|
533 | |
---|
534 | Test Check Condition Conventions |
---|
535 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
---|
536 | |
---|
537 | The following names for test check conditions are used: |
---|
538 | |
---|
539 | eq |
---|
540 | The actual value must equal the expected value. |
---|
541 | |
---|
542 | ne |
---|
543 | The actual value must not equal the value of the second parameter. |
---|
544 | |
---|
545 | ge |
---|
546 | The actual value must be greater than or equal to the expected value. |
---|
547 | |
---|
548 | gt |
---|
549 | The actual value must be greater than the expected value. |
---|
550 | |
---|
551 | le |
---|
552 | The actual value must be less than or equal to the expected value. |
---|
553 | |
---|
554 | lt |
---|
555 | The actual value must be less than the expected value. |
---|
556 | |
---|
557 | If the actual value satisfies the test check condition, then the test check |
---|
558 | passes, otherwise it fails. |
---|
559 | |
---|
560 | Test Check Variant Conventions |
---|
561 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
---|
562 | |
---|
563 | The `T_quiet_*()` test check variants do not increment the test step counter |
---|
564 | and only print a message if the test check fails. This is helpful in case a |
---|
565 | test check appears in a tight loop. |
---|
566 | |
---|
567 | The `T_step_*(step, ...)` test check variants check in addition that the test |
---|
568 | step counter is equal to the specified test step value, otherwise the test |
---|
569 | check fails. |
---|
570 | |
---|
571 | The `T_assert_*()` and `T_step_assert_*(step, ...)` test check variants stop |
---|
572 | the current test case execution if the test check fails. |
---|
573 | |
---|
574 | The following names for test check type variants are used: |
---|
575 | |
---|
576 | ptr |
---|
577 | The test value must be a pointer (`void *`). |
---|
578 | |
---|
579 | mem |
---|
580 | The test value must be a memory area with a specified length. |
---|
581 | |
---|
582 | str |
---|
583 | The test value must be a null byte terminated string. |
---|
584 | |
---|
585 | nstr |
---|
586 | The length of the test value string is limited to a specified maximum. |
---|
587 | |
---|
588 | char |
---|
589 | The test value must be a character (`char`). |
---|
590 | |
---|
591 | schar |
---|
592 | The test value must be a signed character (`signed char`). |
---|
593 | |
---|
594 | uchar |
---|
595 | The test value must be an unsigned character (`unsigned char`). |
---|
596 | |
---|
597 | short |
---|
598 | The test value must be a short integer (`short`). |
---|
599 | |
---|
600 | ushort |
---|
601 | The test value must be an unsigned short integer (`unsigned short`). |
---|
602 | |
---|
603 | int |
---|
604 | The test value must be an integer (`int`). |
---|
605 | |
---|
606 | uint |
---|
607 | The test value must be an unsigned integer (`unsigned int`). |
---|
608 | |
---|
609 | long |
---|
610 | The test value must be a long integer (`long`). |
---|
611 | |
---|
612 | ulong |
---|
613 | The test value must be an unsigned long integer (`unsigned long`). |
---|
614 | |
---|
615 | ll |
---|
616 | The test value must be a long long integer (`long long`). |
---|
617 | |
---|
618 | ull |
---|
619 | The test value must be an unsigned long long integer (`unsigned long long`). |
---|
620 | |
---|
621 | i8 |
---|
622 | The test value must be a signed 8-bit integer (`int8_t`). |
---|
623 | |
---|
624 | u8 |
---|
625 | The test value must be an unsigned 8-bit integer (`uint8_t`). |
---|
626 | |
---|
627 | i16 |
---|
628 | The test value must be a signed 16-bit integer (`int16_t`). |
---|
629 | |
---|
630 | u16 |
---|
631 | The test value must be an unsigned 16-bit integer (`uint16_t`). |
---|
632 | |
---|
633 | i32 |
---|
634 | The test value must be a signed 32-bit integer (`int32_t`). |
---|
635 | |
---|
636 | u32 |
---|
637 | The test value must be an unsigned 32-bit integer (`uint32_t`). |
---|
638 | |
---|
639 | i64 |
---|
640 | The test value must be a signed 64-bit integer (`int64_t`). |
---|
641 | |
---|
642 | u64 |
---|
643 | The test value must be an unsigned 64-bit integer (`uint64_t`). |
---|
644 | |
---|
645 | iptr |
---|
646 | The test value must be of type `intptr_t`. |
---|
647 | |
---|
648 | uptr |
---|
649 | The test value must be of type `uintptr_t`. |
---|
650 | |
---|
651 | ssz |
---|
652 | The test value must be of type `ssize_t`. |
---|
653 | |
---|
654 | sz |
---|
655 | The test value must be of type `size_t`. |
---|
656 | |
---|
657 | Boolean Expressions |
---|
658 | ~~~~~~~~~~~~~~~~~~~ |
---|
659 | |
---|
660 | The following test checks for boolean expressions are available: |
---|
661 | |
---|
662 | .. code-block:: c |
---|
663 | |
---|
664 | void T_true(bool a, const char *fmt, ...); |
---|
665 | void T_assert_true(bool a, const char *fmt, ...); |
---|
666 | void T_quiet_true(bool a, const char *fmt, ...); |
---|
667 | void T_step_true(unsigned int step, bool a, const char *fmt, ...); |
---|
668 | void T_step_assert_true(unsigned int step, bool a, const char *fmt, ...); |
---|
669 | |
---|
670 | void T_false(bool a, const char *fmt, ...); |
---|
671 | void T_assert_false(bool a, const char *fmt, ...); |
---|
672 | void T_quiet_true(bool a, const char *fmt, ...); |
---|
673 | void T_step_true(unsigned int step, bool a, const char *fmt, ...); |
---|
674 | void T_step_assert_true(unsigned int step, bool a, const char *fmt, ...); |
---|
675 | |
---|
676 | The message is only printed in case the test check fails. The format parameter |
---|
677 | is mandatory. |
---|
678 | |
---|
679 | .. code-block:: c |
---|
680 | :caption: Boolean Test Checks Example |
---|
681 | |
---|
682 | #include <t.h> |
---|
683 | |
---|
684 | T_TEST_CASE(example) |
---|
685 | { |
---|
686 | T_true(true, "test passes, no message output"); |
---|
687 | T_true(false, "test fails"); |
---|
688 | T_quiet_true(true, "quiet test passes, no output at all"); |
---|
689 | T_quiet_true(false, "quiet test fails"); |
---|
690 | T_step_true(2, true, "step test passes, no message output"); |
---|
691 | T_step_true(3, false, "step test fails"); |
---|
692 | T_assert_false(true, "this is a format %s", "string"); |
---|
693 | } |
---|
694 | |
---|
695 | .. code-block:: none |
---|
696 | :caption: Boolean Test Checks Report |
---|
697 | |
---|
698 | B:example |
---|
699 | P:0:0:UI1:test-example.c:5 |
---|
700 | F:1:0:UI1:test-example.c:6:test fails |
---|
701 | F:*:0:UI1:test-example.c:8:quiet test fails |
---|
702 | P:2:0:UI1:test-example.c:9 |
---|
703 | F:3:0:UI1:test-example.c:10:step test fails |
---|
704 | F:4:0:UI1:test-example.c:11:this is a format string |
---|
705 | E:example:N:5:F:4 |
---|
706 | |
---|
707 | Generic Types |
---|
708 | ~~~~~~~~~~~~~ |
---|
709 | |
---|
710 | The following test checks for data types with an equality (`==`) or inequality |
---|
711 | (`!=`) operator are available: |
---|
712 | |
---|
713 | .. code-block:: c |
---|
714 | |
---|
715 | void T_eq(T a, T e, const char *fmt, ...); |
---|
716 | void T_assert_eq(T a, T e, const char *fmt, ...); |
---|
717 | void T_quiet_eq(T a, T e, const char *fmt, ...); |
---|
718 | void T_step_eq(unsigned int step, T a, T e, const char *fmt, ...); |
---|
719 | void T_step_assert_eq(unsigned int step, T a, T e, const char *fmt, ...); |
---|
720 | |
---|
721 | void T_ne(T a, T e, const char *fmt, ...); |
---|
722 | void T_assert_ne(T a, T e, const char *fmt, ...); |
---|
723 | void T_quiet_ne(T a, T e, const char *fmt, ...); |
---|
724 | void T_step_ne(unsigned int step, T a, T e, const char *fmt, ...); |
---|
725 | void T_step_assert_ne(unsigned int step, T a, T e, const char *fmt, ...); |
---|
726 | |
---|
727 | The type name `T` specifies an arbitrary type which must support the |
---|
728 | corresponding operator. The message is only printed in case the test check |
---|
729 | fails. The format parameter is mandatory. |
---|
730 | |
---|
731 | Pointers |
---|
732 | ~~~~~~~~ |
---|
733 | |
---|
734 | The following test checks for pointers are available: |
---|
735 | |
---|
736 | .. code-block:: c |
---|
737 | |
---|
738 | void T_eq_ptr(const void *a, const void *e); |
---|
739 | void T_assert_eq_ptr(const void *a, const void *e); |
---|
740 | void T_quiet_eq_ptr(const void *a, const void *e); |
---|
741 | void T_step_eq_ptr(unsigned int step, const void *a, const void *e); |
---|
742 | void T_step_assert_eq_ptr(unsigned int step, const void *a, const void *e); |
---|
743 | |
---|
744 | void T_ne_ptr(const void *a, const void *e); |
---|
745 | void T_assert_ne_ptr(const void *a, const void *e); |
---|
746 | void T_quiet_ne_ptr(const void *a, const void *e); |
---|
747 | void T_step_ne_ptr(unsigned int step, const void *a, const void *e); |
---|
748 | void T_step_assert_ne_ptr(unsigned int step, const void *a, const void *e); |
---|
749 | |
---|
750 | void T_null(const void *a); |
---|
751 | void T_assert_null(const void *a); |
---|
752 | void T_quiet_null(const void *a); |
---|
753 | void T_step_null(unsigned int step, const void *a); |
---|
754 | void T_step_assert_null(unsigned int step, const void *a); |
---|
755 | |
---|
756 | void T_not_null(const void *a); |
---|
757 | void T_assert_not_null(const void *a); |
---|
758 | void T_quiet_not_null(const void *a); |
---|
759 | void T_step_not_null(unsigned int step, const void *a); |
---|
760 | void T_step_assert_not_null(unsigned int step, const void *a); |
---|
761 | |
---|
762 | An automatically generated message is printed in case the test check fails. |
---|
763 | |
---|
764 | Memory Areas |
---|
765 | ~~~~~~~~~~~~ |
---|
766 | |
---|
767 | The following test checks for memory areas are available: |
---|
768 | |
---|
769 | .. code-block:: c |
---|
770 | |
---|
771 | void T_eq_mem(const void *a, const void *e, size_t n); |
---|
772 | void T_assert_eq_mem(const void *a, const void *e, size_t n); |
---|
773 | void T_quiet_eq_mem(const void *a, const void *e, size_t n); |
---|
774 | void T_step_eq_mem(unsigned int step, const void *a, const void *e, size_t n); |
---|
775 | void T_step_assert_eq_mem(unsigned int step, const void *a, const void *e, size_t n); |
---|
776 | |
---|
777 | void T_ne_mem(const void *a, const void *e, size_t n); |
---|
778 | void T_assert_ne_mem(const void *a, const void *e, size_t n); |
---|
779 | void T_quiet_ne_mem(const void *a, const void *e, size_t n); |
---|
780 | void T_step_ne_mem(unsigned int step, const void *a, const void *e, size_t n); |
---|
781 | void T_step_assert_ne_mem(unsigned int step, const void *a, const void *e, size_t n); |
---|
782 | |
---|
783 | The `memcmp()` function is used to compare the memory areas. An automatically |
---|
784 | generated message is printed in case the test check fails. |
---|
785 | |
---|
786 | Strings |
---|
787 | ~~~~~~~ |
---|
788 | |
---|
789 | The following test checks for strings are available: |
---|
790 | |
---|
791 | .. code-block:: c |
---|
792 | |
---|
793 | void T_eq_str(const char *a, const char *e); |
---|
794 | void T_assert_eq_str(const char *a, const char *e); |
---|
795 | void T_quiet_eq_str(const char *a, const char *e); |
---|
796 | void T_step_eq_str(unsigned int step, const char *a, const char *e); |
---|
797 | void T_step_assert_eq_str(unsigned int step, const char *a, const char *e); |
---|
798 | |
---|
799 | void T_ne_str(const char *a, const char *e); |
---|
800 | void T_assert_ne_str(const char *a, const char *e); |
---|
801 | void T_quiet_ne_str(const char *a, const char *e); |
---|
802 | void T_step_ne_str(unsigned int step, const char *a, const char *e); |
---|
803 | void T_step_assert_ne_str(unsigned int step, const char *a, const char *e); |
---|
804 | |
---|
805 | void T_eq_nstr(const char *a, const char *e, size_t n); |
---|
806 | void T_assert_eq_nstr(const char *a, const char *e, size_t n); |
---|
807 | void T_quiet_eq_nstr(const char *a, const char *e, size_t n); |
---|
808 | void T_step_eq_nstr(unsigned int step, const char *a, const char *e, size_t n); |
---|
809 | void T_step_assert_eq_nstr(unsigned int step, const char *a, const char *e, size_t n); |
---|
810 | |
---|
811 | void T_ne_nstr(const char *a, const char *e, size_t n); |
---|
812 | void T_assert_ne_nstr(const char *a, const char *e, size_t n); |
---|
813 | void T_quiet_ne_nstr(const char *a, const char *e, size_t n); |
---|
814 | void T_step_ne_nstr(unsigned int step, const char *a, const char *e, size_t n); |
---|
815 | void T_step_assert_ne_nstr(unsigned int step, const char *a, const char *e, size_t n); |
---|
816 | |
---|
817 | The `strcmp()` and `strncmp()` functions are used to compare the strings. An |
---|
818 | automatically generated message is printed in case the test check fails. |
---|
819 | |
---|
820 | Characters |
---|
821 | ~~~~~~~~~~ |
---|
822 | |
---|
823 | The following test checks for characters (`char`) are available: |
---|
824 | |
---|
825 | .. code-block:: c |
---|
826 | |
---|
827 | void T_eq_char(char a, char e); |
---|
828 | void T_assert_eq_char(char a, char e); |
---|
829 | void T_quiet_eq_char(char a, char e); |
---|
830 | void T_step_eq_char(unsigned int step, char a, char e); |
---|
831 | void T_step_assert_eq_char(unsigned int step, char a, char e); |
---|
832 | |
---|
833 | void T_ne_char(char a, char e); |
---|
834 | void T_assert_ne_char(char a, char e); |
---|
835 | void T_quiet_ne_char(char a, char e); |
---|
836 | void T_step_ne_char(unsigned int step, char a, char e); |
---|
837 | void T_step_assert_ne_char(unsigned int step, char a, char e); |
---|
838 | |
---|
839 | An automatically generated message is printed in case the test check fails. |
---|
840 | |
---|
841 | Integers |
---|
842 | ~~~~~~~~ |
---|
843 | |
---|
844 | The following test checks for integers are available: |
---|
845 | |
---|
846 | .. code-block:: c |
---|
847 | |
---|
848 | void T_eq_xyz(I a, I e); |
---|
849 | void T_assert_eq_xyz(I a, I e); |
---|
850 | void T_quiet_eq_xyz(I a, I e); |
---|
851 | void T_step_eq_xyz(unsigned int step, I a, I e); |
---|
852 | void T_step_assert_eq_xyz(unsigned int step, I a, I e); |
---|
853 | |
---|
854 | void T_ne_xyz(I a, I e); |
---|
855 | void T_assert_ne_xyz(I a, I e); |
---|
856 | void T_quiet_ne_xyz(I a, I e); |
---|
857 | void T_step_ne_xyz(unsigned int step, I a, I e); |
---|
858 | void T_step_assert_ne_xyz(unsigned int step, I a, I e); |
---|
859 | |
---|
860 | void T_ge_xyz(I a, I e); |
---|
861 | void T_assert_ge_xyz(I a, I e); |
---|
862 | void T_quiet_ge_xyz(I a, I e); |
---|
863 | void T_step_ge_xyz(unsigned int step, I a, I e); |
---|
864 | void T_step_assert_ge_xyz(unsigned int step, I a, I e); |
---|
865 | |
---|
866 | void T_gt_xyz(I a, I e); |
---|
867 | void T_assert_gt_xyz(I a, I e); |
---|
868 | void T_quiet_gt_xyz(I a, I e); |
---|
869 | void T_step_gt_xyz(unsigned int step, I a, I e); |
---|
870 | void T_step_assert_gt_xyz(unsigned int step, I a, I e); |
---|
871 | |
---|
872 | void T_le_xyz(I a, I e); |
---|
873 | void T_assert_le_xyz(I a, I e); |
---|
874 | void T_quiet_le_xyz(I a, I e); |
---|
875 | void T_step_le_xyz(unsigned int step, I a, I e); |
---|
876 | void T_step_assert_le_xyz(unsigned int step, I a, I e); |
---|
877 | |
---|
878 | void T_lt_xyz(I a, I e); |
---|
879 | void T_assert_lt_xyz(I a, I e); |
---|
880 | void T_quiet_lt_xyz(I a, I e); |
---|
881 | void T_step_lt_xyz(unsigned int step, I a, I e); |
---|
882 | void T_step_assert_lt_xyz(unsigned int step, I a, I e); |
---|
883 | |
---|
884 | The type variant `xyz` must be `schar`, `uchar`, `short`, `ushort`, `int`, |
---|
885 | `uint`, `long`, `ulong`, `ll`, `ull`, `i8`, `u8`, `i16`, `u16`, `i32`, `u32`, |
---|
886 | `i64`, `u64`, `iptr`, `uptr`, `ssz`, or `sz`. |
---|
887 | |
---|
888 | The type name `I` must be compatible to the type variant. |
---|
889 | |
---|
890 | An automatically generated message is printed in case the test check fails. |
---|
891 | |
---|
892 | RTEMS Status Codes |
---|
893 | ~~~~~~~~~~~~~~~~~~ |
---|
894 | |
---|
895 | The following test checks for RTEMS status codes are available: |
---|
896 | |
---|
897 | .. code-block:: c |
---|
898 | |
---|
899 | void T_rsc(rtems_status_code a, rtems_status_code e); |
---|
900 | void T_assert_rsc(rtems_status_code a, rtems_status_code e); |
---|
901 | void T_quiet_rsc(rtems_status_code a, rtems_status_code e); |
---|
902 | void T_step_rsc(unsigned int step, rtems_status_code a, rtems_status_code e); |
---|
903 | void T_step_assert_rsc(unsigned int step, rtems_status_code a, rtems_status_code e); |
---|
904 | |
---|
905 | void T_rsc_success(rtems_status_code a); |
---|
906 | void T_assert_rsc_success(rtems_status_code a); |
---|
907 | void T_quiet_rsc_success(rtems_status_code a); |
---|
908 | void T_step_rsc_success(unsigned int step, rtems_status_code a); |
---|
909 | void T_step_assert_rsc_success(unsigned int step, rtems_status_code a); |
---|
910 | |
---|
911 | An automatically generated message is printed in case the test check fails. |
---|
912 | |
---|
913 | POSIX Error Numbers |
---|
914 | ~~~~~~~~~~~~~~~~~~~ |
---|
915 | |
---|
916 | The following test checks for POSIX error numbers are available: |
---|
917 | |
---|
918 | .. code-block:: c |
---|
919 | |
---|
920 | void T_eno(int a, int e); |
---|
921 | void T_assert_eno(int a, int e); |
---|
922 | void T_quiet_eno(int a, int e); |
---|
923 | void T_step_eno(unsigned int step, int a, int e); |
---|
924 | void T_step_assert_eno(unsigned int step, int a, int e); |
---|
925 | |
---|
926 | void T_eno_success(int a); |
---|
927 | void T_assert_eno_success(int a); |
---|
928 | void T_quiet_eno_success(int a); |
---|
929 | void T_step_eno_success(unsigned int step, int a); |
---|
930 | void T_step_assert_eno_success(unsigned int step, int a); |
---|
931 | |
---|
932 | The actual and expected value must be a POSIX error number, e.g. EINVAL, |
---|
933 | ENOMEM, etc. An automatically generated message is printed in case the test |
---|
934 | check fails. |
---|
935 | |
---|
936 | POSIX Status Codes |
---|
937 | ~~~~~~~~~~~~~~~~~~ |
---|
938 | |
---|
939 | The following test checks for POSIX status codes are available: |
---|
940 | |
---|
941 | .. code-block:: c |
---|
942 | |
---|
943 | void T_psx_error(int a, int eno); |
---|
944 | void T_assert_psx_error(int a, int eno); |
---|
945 | void T_quiet_psx_error(int a, int eno); |
---|
946 | void T_step_psx_error(unsigned int step, int a, int eno); |
---|
947 | void T_step_assert_psx_error(unsigned int step, int a, int eno); |
---|
948 | |
---|
949 | void T_psx_success(int a); |
---|
950 | void T_assert_psx_success(int a); |
---|
951 | void T_quiet_psx_success(int a); |
---|
952 | void T_step_psx_success(unsigned int step, int a); |
---|
953 | void T_step_assert_psx_success(unsigned int step, int a); |
---|
954 | |
---|
955 | The `eno` value must be a POSIX error number, e.g. EINVAL, ENOMEM, etc. An |
---|
956 | actual value of zero indicates success. An actual value of minus one indicates |
---|
957 | an error. An automatically generated message is printed in case the test check |
---|
958 | fails. |
---|
959 | |
---|
960 | .. code-block:: c |
---|
961 | :caption: POSIX Status Code Example |
---|
962 | |
---|
963 | #include <t.h> |
---|
964 | |
---|
965 | #include <sys/stat.h> |
---|
966 | #include <errno.h> |
---|
967 | |
---|
968 | T_TEST_CASE(stat) |
---|
969 | { |
---|
970 | struct stat st; |
---|
971 | int status; |
---|
972 | |
---|
973 | errno = 0; |
---|
974 | status = stat("foobar", &st); |
---|
975 | T_psx_error(status, ENOENT); |
---|
976 | } |
---|
977 | |
---|
978 | .. code-block:: none |
---|
979 | :caption: POSIX Status Code Report |
---|
980 | |
---|
981 | B:stat |
---|
982 | P:0:0:UI1:test-psx.c:13 |
---|
983 | E:stat:N:1:F:0 |
---|
984 | |
---|
985 | Custom Log Messages |
---|
986 | ------------------- |
---|
987 | |
---|
988 | You can print custom log messages with the `T_log()` function: |
---|
989 | |
---|
990 | .. code-block:: c |
---|
991 | |
---|
992 | void T_log(T_verbosity verbosity, char const *fmt, ...); |
---|
993 | |
---|
994 | A newline is automatically added to terminate the log message line. |
---|
995 | |
---|
996 | .. code-block:: c |
---|
997 | :caption: Custom Log Message Example |
---|
998 | |
---|
999 | #include <t.h> |
---|
1000 | |
---|
1001 | T_TEST_CASE(log) |
---|
1002 | { |
---|
1003 | T_log(T_NORMAL, "a custom message %i, %i, %i", 1, 2, 3); |
---|
1004 | T_set_verbosity(T_QUIET); |
---|
1005 | T_log(T_NORMAL, "not verbose enough"); |
---|
1006 | } |
---|
1007 | |
---|
1008 | .. code-block:: none |
---|
1009 | :caption: Custom Log Message Report |
---|
1010 | |
---|
1011 | B:log |
---|
1012 | L:a custom message 1, 2, 3 |
---|
1013 | E:log:N:0:F:0 |
---|
1014 | |
---|
1015 | Time Services |
---|
1016 | ------------- |
---|
1017 | |
---|
1018 | The test framework provides two unsigned integer types for time values. The |
---|
1019 | `T_ticks` unsigned integer type is used by the `T_tick()` function which |
---|
1020 | measures time using the highest frequency counter available on the platform. |
---|
1021 | It should only be used to measure small time intervals. The `T_time` unsigned |
---|
1022 | integer type is used by the `T_now()` function which returns the current |
---|
1023 | monotonic clock value of the platform, e.g. `CLOCK_MONOTONIC`. |
---|
1024 | |
---|
1025 | .. code-block:: c |
---|
1026 | |
---|
1027 | T_ticks T_tick(void); |
---|
1028 | |
---|
1029 | T_time T_now(void); |
---|
1030 | |
---|
1031 | The reference time point for these two clocks is unspecified. You can obtain |
---|
1032 | the test case begin time with the `T_case_begin_time()` function. |
---|
1033 | |
---|
1034 | .. code-block:: c |
---|
1035 | |
---|
1036 | T_time T_case_begin_time(void); |
---|
1037 | |
---|
1038 | You can convert time into ticks with the `T_time_to_ticks()` function and vice |
---|
1039 | versa with the `T_ticks_to_time()` function. |
---|
1040 | |
---|
1041 | .. code-block:: c |
---|
1042 | |
---|
1043 | T_time T_ticks_to_time(T_ticks ticks); |
---|
1044 | |
---|
1045 | T_ticks T_time_to_ticks(T_time time); |
---|
1046 | |
---|
1047 | You can convert seconds and nanoseconds values into a combined time value with |
---|
1048 | the `T_seconds_and_nanoseconds_to_time()` function. You can convert a time |
---|
1049 | value into separate seconds and nanoseconds values with the |
---|
1050 | `T_time_to_seconds_and_nanoseconds()` function. |
---|
1051 | |
---|
1052 | .. code-block:: c |
---|
1053 | |
---|
1054 | T_time T_seconds_and_nanoseconds_to_time(uint32_t s, uint32_t ns); |
---|
1055 | |
---|
1056 | void T_time_to_seconds_and_nanoseconds(T_time time, uint32_t *s, uint32_t *ns); |
---|
1057 | |
---|
1058 | You can convert a time value into a string represention. The time unit of the |
---|
1059 | string representation is seconds. The precision of the string represention may |
---|
1060 | be nanoseconds, microseconds, milliseconds, or seconds. You have to provide a |
---|
1061 | buffer for the string (`T_time_string`). |
---|
1062 | |
---|
1063 | .. code-block:: c |
---|
1064 | |
---|
1065 | const char *T_time_to_string_ns(T_time time, T_time_string buffer); |
---|
1066 | |
---|
1067 | const char *T_time_to_string_us(T_time time, T_time_string buffer); |
---|
1068 | |
---|
1069 | const char *T_time_to_string_ms(T_time time, T_time_string buffer); |
---|
1070 | |
---|
1071 | const char *T_time_to_string_s(T_time time, T_time_string buffer); |
---|
1072 | |
---|
1073 | .. code-block:: c |
---|
1074 | :caption: Time String Example |
---|
1075 | |
---|
1076 | #include <t.h> |
---|
1077 | |
---|
1078 | T_TEST_CASE(time_to_string) |
---|
1079 | { |
---|
1080 | T_time_string ts; |
---|
1081 | T_time t; |
---|
1082 | uint32_t s; |
---|
1083 | uint32_t ns; |
---|
1084 | |
---|
1085 | t = T_seconds_and_nanoseconds_to_time(0, 123456789); |
---|
1086 | T_eq_str(T_time_to_string_ns(t, ts), "0.123456789"); |
---|
1087 | T_eq_str(T_time_to_string_us(t, ts), "0.123456"); |
---|
1088 | T_eq_str(T_time_to_string_ms(t, ts), "0.123"); |
---|
1089 | T_eq_str(T_time_to_string_s(t, ts), "0"); |
---|
1090 | |
---|
1091 | T_time_to_seconds_and_nanoseconds(t, &s, &ns); |
---|
1092 | T_eq_u32(s, 0); |
---|
1093 | T_eq_u32(ns, 123456789); |
---|
1094 | } |
---|
1095 | |
---|
1096 | .. code-block:: none |
---|
1097 | :caption: Time String Report |
---|
1098 | |
---|
1099 | B:time_to_string |
---|
1100 | P:0:0:UI1:test-time.c:11 |
---|
1101 | P:1:0:UI1:test-time.c:12 |
---|
1102 | P:2:0:UI1:test-time.c:13 |
---|
1103 | P:3:0:UI1:test-time.c:14 |
---|
1104 | P:4:0:UI1:test-time.c:17 |
---|
1105 | P:5:0:UI1:test-time.c:18 |
---|
1106 | E:time_to_string:N:6:F:0:D:0.005250 |
---|
1107 | |
---|
1108 | You can convert a tick value into a string represention. The time unit of the |
---|
1109 | string representation is seconds. The precision of the string represention may |
---|
1110 | be nanoseconds, microseconds, milliseconds, or seconds. You have to provide a |
---|
1111 | buffer for the string (`T_time_string`). |
---|
1112 | |
---|
1113 | .. code-block:: c |
---|
1114 | |
---|
1115 | const char *T_ticks_to_string_ns(T_ticks ticks, T_time_string buffer); |
---|
1116 | |
---|
1117 | const char *T_ticks_to_string_us(T_ticks ticks, T_time_string buffer); |
---|
1118 | |
---|
1119 | const char *T_ticks_to_string_ms(T_ticks ticks, T_time_string buffer); |
---|
1120 | |
---|
1121 | const char *T_ticks_to_string_s(T_ticks ticks, T_time_string buffer); |
---|
1122 | |
---|
1123 | Code Runtime Measurements |
---|
1124 | ------------------------- |
---|
1125 | |
---|
1126 | You can measure the runtime of code fragments in several execution environment |
---|
1127 | variants with the `T_measure_runtime()` function. This function needs a |
---|
1128 | context which must be created with the `T_measure_runtime_create()` function. |
---|
1129 | The context is automatically destroyed after the test case execution. |
---|
1130 | |
---|
1131 | .. code-block:: c |
---|
1132 | |
---|
1133 | typedef struct { |
---|
1134 | size_t sample_count; |
---|
1135 | } T_measure_runtime_config; |
---|
1136 | |
---|
1137 | typedef struct { |
---|
1138 | const char *name; |
---|
1139 | int flags; |
---|
1140 | void (*setup)(void *arg); |
---|
1141 | void (*body)(void *arg); |
---|
1142 | bool (*teardown)(void *arg, T_ticks *delta, uint32_t tic, uint32_t toc, |
---|
1143 | unsigned int retry); |
---|
1144 | void *arg; |
---|
1145 | } T_measure_runtime_request; |
---|
1146 | |
---|
1147 | T_measure_runtime_context *T_measure_runtime_create( |
---|
1148 | const T_measure_runtime_config *config); |
---|
1149 | |
---|
1150 | void T_measure_runtime(T_measure_runtime_context *ctx, |
---|
1151 | const T_measure_runtime_request *request); |
---|
1152 | |
---|
1153 | The runtime measurement is performed for the `body` request handler of the |
---|
1154 | measurement request (`T_measure_runtime_request`). The optional `setup` |
---|
1155 | request handler is called before each invocation of the `body` request handler. |
---|
1156 | The optional `teardown` request handler is called after each invocation of the |
---|
1157 | `body` request handler. It has several parameters and a return status. If it |
---|
1158 | returns true, then this measurement sample value is recorded, otherwise the |
---|
1159 | measurement is retried. The `delta` parameter is the current measurement |
---|
1160 | sample value. It can be altered by the `teardown` request handler. The `tic` |
---|
1161 | and `toc` parameters are the system tick values before and after the request |
---|
1162 | body invocation. The `retry` parameter is the current retry counter. The |
---|
1163 | runtime of the operational `setup` and `teardown` request handlers is not |
---|
1164 | measured. |
---|
1165 | |
---|
1166 | You can control some aspects of the measurement through the request flags (use |
---|
1167 | zero for the default): |
---|
1168 | |
---|
1169 | T_MEASURE_RUNTIME_ALLOW_CLOCK_ISR |
---|
1170 | Allow clock interrupts during the measurement. By default, measurements |
---|
1171 | during which a clock interrupt happened are discarded unless it happens two |
---|
1172 | times in a row. |
---|
1173 | |
---|
1174 | T_MEASURE_RUNTIME_REPORT_SAMPLES |
---|
1175 | Report all measurement samples. |
---|
1176 | |
---|
1177 | T_MEASURE_RUNTIME_DISABLE_VALID_CACHE |
---|
1178 | Disable the `ValidCache` execution environment variant. |
---|
1179 | |
---|
1180 | T_MEASURE_RUNTIME_DISABLE_HOT_CACHE |
---|
1181 | Disable the `HotCache` execution environment variant. |
---|
1182 | |
---|
1183 | T_MEASURE_RUNTIME_DISABLE_DIRTY_CACHE |
---|
1184 | Disable the `DirtyCache` execution environment variant. |
---|
1185 | |
---|
1186 | T_MEASURE_RUNTIME_DISABLE_MINOR_LOAD |
---|
1187 | Disable the `Load` execution environment variants with a load worker count |
---|
1188 | less than the processor count. |
---|
1189 | |
---|
1190 | T_MEASURE_RUNTIME_DISABLE_MAX_LOAD |
---|
1191 | Disable the `Load` execution environment variant with a load worker count |
---|
1192 | equal to the processor count. |
---|
1193 | |
---|
1194 | The execution environment variants (`M:V`) are: |
---|
1195 | |
---|
1196 | ValidCache |
---|
1197 | Before the `body` request handler is invoked a memory area with twice the |
---|
1198 | size of the outer-most data cache is completely read. This fills the data |
---|
1199 | cache with valid cache lines which are unrelated to the `body` request |
---|
1200 | handler. |
---|
1201 | |
---|
1202 | You can disable this variant with the |
---|
1203 | `T_MEASURE_RUNTIME_DISABLE_VALID_CACHE` request flag. |
---|
1204 | |
---|
1205 | HotCache |
---|
1206 | Before the `body` request handler is invoked the `body` request handler is |
---|
1207 | called without measuring the runtime. The aim is to load all data used by |
---|
1208 | the `body` request handler to the cache. |
---|
1209 | |
---|
1210 | You can disable this variant with the |
---|
1211 | `T_MEASURE_RUNTIME_DISABLE_HOT_CACHE` request flag. |
---|
1212 | |
---|
1213 | DirtyCache |
---|
1214 | Before the `body` request handler is invoked a memory area with twice the |
---|
1215 | size of the outer-most data cache is completely written with new data. |
---|
1216 | This should produce a data cache with dirty cache lines which are unrelated |
---|
1217 | to the `body` request handler. In addition, the entire instruction cache |
---|
1218 | is invalidated. |
---|
1219 | |
---|
1220 | You can disable this variant with the |
---|
1221 | `T_MEASURE_RUNTIME_DISABLE_DIRTY_CACHE` request flag. |
---|
1222 | |
---|
1223 | Load |
---|
1224 | This variant tries to get close to worst-case conditions. The cache is set |
---|
1225 | up according to the `DirtyCache` variant. In addition, other processors |
---|
1226 | try to fully load the memory system. The load is produced through writes |
---|
1227 | to a memory area with twice the size of the outer-most data cache. The |
---|
1228 | load variant is performed multiple times with a different set of active |
---|
1229 | load worker threads (`M:L`). The active workers range from one up to the |
---|
1230 | processor count. |
---|
1231 | |
---|
1232 | You can disable these variants with the |
---|
1233 | `T_MEASURE_RUNTIME_DISABLE_MINOR_LOAD` and |
---|
1234 | `T_MEASURE_RUNTIME_DISABLE_MAX_LOAD` request flags. |
---|
1235 | |
---|
1236 | On SPARC, the `body` request handler is called with a register window |
---|
1237 | setting so that window overflow traps will occur in the next level function |
---|
1238 | call. |
---|
1239 | |
---|
1240 | Each execution in an environment variant produces a sample set of `body` |
---|
1241 | request handler runtime measurements. The minimum (`M:MI`), first quartile |
---|
1242 | (`M:Q1`), median (`M:Q2`), third quartile (`M:Q3`), maximum (`M:MX`), median |
---|
1243 | absolute deviation (`M:MAD`), and the sum of the sample values (`M:D`) is |
---|
1244 | reported. |
---|
1245 | |
---|
1246 | .. code-block:: c |
---|
1247 | :caption: Code Runtime Measurement Example |
---|
1248 | |
---|
1249 | #include <t.h> |
---|
1250 | |
---|
1251 | static void |
---|
1252 | empty(void *arg) |
---|
1253 | { |
---|
1254 | (void)arg; |
---|
1255 | } |
---|
1256 | |
---|
1257 | T_TEST_CASE(measure_empty) |
---|
1258 | { |
---|
1259 | static const T_measure_runtime_config config = { |
---|
1260 | .sample_count = 1024 |
---|
1261 | }; |
---|
1262 | T_measure_runtime_context *ctx; |
---|
1263 | T_measure_runtime_request req; |
---|
1264 | |
---|
1265 | ctx = T_measure_runtime_create(&config); |
---|
1266 | T_assert_not_null(ctx); |
---|
1267 | |
---|
1268 | memset(&req, 0, sizeof(req)); |
---|
1269 | req.name = "Empty"; |
---|
1270 | req.body = empty; |
---|
1271 | T_measure_runtime(ctx, &req); |
---|
1272 | } |
---|
1273 | |
---|
1274 | .. code-block:: none |
---|
1275 | :caption: Code Runtime Measurement Report |
---|
1276 | |
---|
1277 | B:measure_empty |
---|
1278 | P:0:0:UI1:test-rtems-measure.c:18 |
---|
1279 | M:B:Empty |
---|
1280 | M:V:ValidCache |
---|
1281 | M:N:1024 |
---|
1282 | M:MI:0.000000000 |
---|
1283 | M:Q1:0.000000000 |
---|
1284 | M:Q2:0.000000000 |
---|
1285 | M:Q3:0.000000000 |
---|
1286 | M:MX:0.000000009 |
---|
1287 | M:MAD:0.000000000 |
---|
1288 | M:D:0.000000485 |
---|
1289 | M:E:Empty:D:0.208984183 |
---|
1290 | M:B:Empty |
---|
1291 | M:V:HotCache |
---|
1292 | M:N:1024 |
---|
1293 | M:MI:0.000000003 |
---|
1294 | M:Q1:0.000000003 |
---|
1295 | M:Q2:0.000000003 |
---|
1296 | M:Q3:0.000000003 |
---|
1297 | M:MX:0.000000006 |
---|
1298 | M:MAD:0.000000000 |
---|
1299 | M:D:0.000002626 |
---|
1300 | M:E:Empty:D:0.000017046 |
---|
1301 | M:B:Empty |
---|
1302 | M:V:DirtyCache |
---|
1303 | M:N:1024 |
---|
1304 | M:MI:0.000000007 |
---|
1305 | M:Q1:0.000000007 |
---|
1306 | M:Q2:0.000000007 |
---|
1307 | M:Q3:0.000000008 |
---|
1308 | M:MX:0.000000559 |
---|
1309 | M:MAD:0.000000000 |
---|
1310 | M:D:0.000033244 |
---|
1311 | M:E:Empty:D:1.887834875 |
---|
1312 | M:B:Empty |
---|
1313 | M:V:Load |
---|
1314 | M:L:1 |
---|
1315 | M:N:1024 |
---|
1316 | M:MI:0.000000000 |
---|
1317 | M:Q1:0.000000002 |
---|
1318 | M:Q2:0.000000002 |
---|
1319 | M:Q3:0.000000003 |
---|
1320 | M:MX:0.000000288 |
---|
1321 | M:MAD:0.000000000 |
---|
1322 | M:D:0.000002421 |
---|
1323 | M:E:Empty:D:0.001798809 |
---|
1324 | [... 22 more load variants ...] |
---|
1325 | M:E:Empty:D:0.021252583 |
---|
1326 | M:B:Empty |
---|
1327 | M:V:Load |
---|
1328 | M:L:24 |
---|
1329 | M:N:1024 |
---|
1330 | M:MI:0.000000001 |
---|
1331 | M:Q1:0.000000002 |
---|
1332 | M:Q2:0.000000002 |
---|
1333 | M:Q3:0.000000003 |
---|
1334 | M:MX:0.000001183 |
---|
1335 | M:MAD:0.000000000 |
---|
1336 | M:D:0.000003406 |
---|
1337 | M:E:Empty:D:0.015188063 |
---|
1338 | E:measure_empty:N:1:F:0:D:14.284869 |
---|
1339 | |
---|
1340 | |
---|
1341 | Test Runner |
---|
1342 | ----------- |
---|
1343 | |
---|
1344 | You can call the `T_main()` function to run all registered test cases. |
---|
1345 | |
---|
1346 | .. code-block:: c |
---|
1347 | |
---|
1348 | int T_main(const T_config *config); |
---|
1349 | |
---|
1350 | The `T_main()` function returns 0 if all test cases passed, otherwise it |
---|
1351 | returns 1. Concurrent execution of the `T_main()` function is undefined |
---|
1352 | behaviour. |
---|
1353 | |
---|
1354 | You can ask if you execute within the context of the test runner with the |
---|
1355 | `T_is_runner()` function: |
---|
1356 | |
---|
1357 | .. code-block:: c |
---|
1358 | |
---|
1359 | bool T_is_runner(void); |
---|
1360 | |
---|
1361 | It returns `true` if you execute within the context of the test runner (the |
---|
1362 | context which executes for example `T_main()`). Otherwise it returns `false`, |
---|
1363 | for example if you execute in another task, in interrupt context, nobody |
---|
1364 | executes `T_main()`, or during system initialization on another processor. |
---|
1365 | |
---|
1366 | On RTEMS, you have to register the test cases with the `T_register()` function |
---|
1367 | before you call `T_main()`. This makes it possible to run low level tests, for |
---|
1368 | example without the operating system directly in `boot_card()` or during device |
---|
1369 | driver initialization. On other platforms, the `T_register()` is a no |
---|
1370 | operation. |
---|
1371 | |
---|
1372 | .. code-block:: c |
---|
1373 | |
---|
1374 | void T_register(void); |
---|
1375 | |
---|
1376 | You can run test cases also individually. Use `T_run_initialize()` to |
---|
1377 | initialize the test runner. Call `T_run_all()` to run all or `T_run_by_name()` |
---|
1378 | to run specific registered test cases. Call `T_case_begin()` to begin a |
---|
1379 | freestanding test case and call `T_case_end()` to finish it. Finally, |
---|
1380 | call `T_run_finalize()`. |
---|
1381 | |
---|
1382 | .. code-block:: c |
---|
1383 | |
---|
1384 | void T_run_initialize(const T_config *config); |
---|
1385 | |
---|
1386 | void T_run_all(void); |
---|
1387 | |
---|
1388 | void T_run_by_name(const char *name); |
---|
1389 | |
---|
1390 | void T_case_begin(const char *name, const T_fixture *fixture); |
---|
1391 | |
---|
1392 | void T_case_end(void); |
---|
1393 | |
---|
1394 | bool T_run_finalize(void); |
---|
1395 | |
---|
1396 | The `T_run_finalize()` function returns `true` if all test cases passed, |
---|
1397 | otherwise it returns `false`. Concurrent execution of the runner functions |
---|
1398 | (including `T_main()`) is undefined behaviour. The test suite configuration |
---|
1399 | must be persistent throughout the test run. |
---|
1400 | |
---|
1401 | .. code-block:: c |
---|
1402 | |
---|
1403 | typedef enum { |
---|
1404 | T_EVENT_RUN_INITIALIZE, |
---|
1405 | T_EVENT_CASE_EARLY, |
---|
1406 | T_EVENT_CASE_BEGIN, |
---|
1407 | T_EVENT_CASE_END, |
---|
1408 | T_EVENT_CASE_LATE, |
---|
1409 | T_EVENT_RUN_FINALIZE |
---|
1410 | } T_event; |
---|
1411 | |
---|
1412 | typedef void (*T_action)(T_event, const char *); |
---|
1413 | |
---|
1414 | typedef void (*T_putchar)(int, void *); |
---|
1415 | |
---|
1416 | typedef struct { |
---|
1417 | const char *name; |
---|
1418 | T_putchar putchar; |
---|
1419 | void *putchar_arg; |
---|
1420 | T_verbosity verbosity; |
---|
1421 | T_time (*now)(void); |
---|
1422 | size_t action_count; |
---|
1423 | const T_action *actions; |
---|
1424 | } T_config; |
---|
1425 | |
---|
1426 | With the test suite configuration you can specifiy the test suite name, the put |
---|
1427 | character handler used the output the test report, the initial verbosity, the |
---|
1428 | monotonic time provider and an optional set of test suite actions. The test |
---|
1429 | suite actions are called with the test suite name for test suite run events |
---|
1430 | (`T_EVENT_RUN_INITIALIZE` and `T_EVENT_RUN_FINALIZE`) and the test case name |
---|
1431 | for the test case events (`T_EVENT_CASE_EARLY`, `T_EVENT_CASE_BEGIN`, |
---|
1432 | `T_EVENT_CASE_END` and `T_EVENT_CASE_LATE`). |
---|
1433 | |
---|
1434 | Test Verbosity |
---|
1435 | -------------- |
---|
1436 | |
---|
1437 | Three test verbosity levels are defined: |
---|
1438 | |
---|
1439 | T_QUIET |
---|
1440 | Only the test suite begin, system, test case end, and test suite end lines |
---|
1441 | are printed. |
---|
1442 | |
---|
1443 | T_NORMAL |
---|
1444 | Prints everything except passed test lines. |
---|
1445 | |
---|
1446 | T_VERBOSE |
---|
1447 | Prints everything. |
---|
1448 | |
---|
1449 | The test verbosity level can be set within the scope of one test case with the |
---|
1450 | `T_set_verbosity()` function: |
---|
1451 | |
---|
1452 | .. code-block:: c |
---|
1453 | |
---|
1454 | T_verbosity T_set_verbosity(T_verbosity new_verbosity); |
---|
1455 | |
---|
1456 | The function returns the previous verbosity. After the test case, the |
---|
1457 | configured verbosity is automatically restored. |
---|
1458 | |
---|
1459 | An example with `T_QUIET` verbosity: |
---|
1460 | |
---|
1461 | .. code-block:: none |
---|
1462 | |
---|
1463 | A:xyz |
---|
1464 | S:Platform:RTEMS |
---|
1465 | [...] |
---|
1466 | E:a:N:2:F:1 |
---|
1467 | E:b:N:0:F:1 |
---|
1468 | E:c:N:1:F:1 |
---|
1469 | E:d:N:6:F:0 |
---|
1470 | Z:xyz:C:4:N:9:F:3 |
---|
1471 | |
---|
1472 | The same example with `T_NORMAL` verbosity: |
---|
1473 | |
---|
1474 | .. code-block:: none |
---|
1475 | |
---|
1476 | A:xyz |
---|
1477 | S:Platform:RTEMS |
---|
1478 | [...] |
---|
1479 | B:a |
---|
1480 | F:1:0:UI1:test-verbosity.c:6:test fails |
---|
1481 | E:a:N:2:F:1 |
---|
1482 | B:b |
---|
1483 | F:*:0:UI1:test-verbosity.c:12:quiet test fails |
---|
1484 | E:b:N:0:F:1 |
---|
1485 | B:c |
---|
1486 | F:0:0:UI1:test-verbosity.c:17:this is a format string |
---|
1487 | E:c:N:1:F:1 |
---|
1488 | B:d |
---|
1489 | E:d:N:6:F:0 |
---|
1490 | Z:xyz:C:4:N:9:F:3 |
---|
1491 | |
---|
1492 | The same example with `T_VERBOSE` verbosity: |
---|
1493 | |
---|
1494 | .. code-block:: none |
---|
1495 | |
---|
1496 | A:xyz |
---|
1497 | S:Platform:RTEMS |
---|
1498 | [...] |
---|
1499 | B:a |
---|
1500 | P:0:0:UI1:test-verbosity.c:5 |
---|
1501 | F:1:0:UI1:test-verbosity.c:6:test fails |
---|
1502 | E:a:N:2:F:1 |
---|
1503 | B:b |
---|
1504 | F:*:0:UI1:test-verbosity.c:12:quiet test fails |
---|
1505 | E:b:N:0:F:1 |
---|
1506 | B:c |
---|
1507 | F:0:0:UI1:test-verbosity.c:17:this is a format string |
---|
1508 | E:c:N:1:F:1 |
---|
1509 | B:d |
---|
1510 | P:0:0:UI1:test-verbosity.c:22 |
---|
1511 | P:1:0:UI1:test-verbosity.c:23 |
---|
1512 | P:2:0:UI1:test-verbosity.c:24 |
---|
1513 | P:3:0:UI1:test-verbosity.c:25 |
---|
1514 | P:4:0:UI1:test-verbosity.c:26 |
---|
1515 | P:5:0:UI1:test-verbosity.c:27 |
---|
1516 | E:d:N:6:F:0 |
---|
1517 | Z:xyz:C:4:N:9:F:3 |
---|
1518 | |
---|
1519 | .. _RTEMSTestFrameworkTestReporting: |
---|
1520 | |
---|
1521 | Test Reporting |
---|
1522 | -------------- |
---|
1523 | |
---|
1524 | The test reporting is line based which should be easy to parse with a simple |
---|
1525 | state machine. Each line consists of a set of fields separated by colon |
---|
1526 | characters (`:`). The first character of the line determines the line format: |
---|
1527 | |
---|
1528 | A |
---|
1529 | A test suite begin line. It has the format: |
---|
1530 | |
---|
1531 | **A:<TestSuite>** |
---|
1532 | |
---|
1533 | A description of the field follows: |
---|
1534 | |
---|
1535 | <TestSuite> |
---|
1536 | The test suite name. Must not contain colon characters (`:`). |
---|
1537 | |
---|
1538 | S |
---|
1539 | A test suite system line. It has the format: |
---|
1540 | |
---|
1541 | **S:<Key>:<Value>** |
---|
1542 | |
---|
1543 | A description of the fields follows: |
---|
1544 | |
---|
1545 | <Key> |
---|
1546 | A key string. Must not contain colon characters (`:`). |
---|
1547 | |
---|
1548 | <Value> |
---|
1549 | An arbitrary key value string. May contain colon characters (`:`). |
---|
1550 | |
---|
1551 | B |
---|
1552 | A test case begin line. It has the format: |
---|
1553 | |
---|
1554 | **B:<TestCase>** |
---|
1555 | |
---|
1556 | A description of the field follows: |
---|
1557 | |
---|
1558 | <TestCase> |
---|
1559 | A test case name. Must not contain colon characters (`:`). |
---|
1560 | |
---|
1561 | P |
---|
1562 | A test pass line. It has the format: |
---|
1563 | |
---|
1564 | **P:<Step>:<Processor>:<Task>:<File>:<Line>** |
---|
1565 | |
---|
1566 | A description of the fields follows: |
---|
1567 | |
---|
1568 | <Step> |
---|
1569 | Each non-quiet test has a unique test step counter value in each test case |
---|
1570 | execution. The test step counter is set to zero before the test case |
---|
1571 | executes. For quiet test checks, there is no associated test step and the |
---|
1572 | character `*` instead of an integer is used to indicate this. |
---|
1573 | |
---|
1574 | <Processor> |
---|
1575 | The processor index of the processor which executed at least one |
---|
1576 | instruction of the corresponding test. |
---|
1577 | |
---|
1578 | <Task> |
---|
1579 | The name of the task which executed the corresponding test if the test |
---|
1580 | executed in task context. The name `ISR` indicates that the test executed |
---|
1581 | in interrupt context. The name `?` indicates that the test executed in an |
---|
1582 | arbitrary context with no valid executing task. |
---|
1583 | |
---|
1584 | <File> |
---|
1585 | The name of the source file which contains the corresponding test. A |
---|
1586 | source file of `*` indicates that no test source file is associated |
---|
1587 | with the test, e.g. it was produced by the test framework itself. |
---|
1588 | |
---|
1589 | <Line> |
---|
1590 | The line of the test statement in the source file which contains the |
---|
1591 | corresponding test. A line number of `*` indicates that no test source |
---|
1592 | file is associated with the test, e.g. it was produced by the test |
---|
1593 | framework itself. |
---|
1594 | |
---|
1595 | F |
---|
1596 | A test failure line. It has the format: |
---|
1597 | |
---|
1598 | **F:<Step>:<Processor>:<Task>:<File>:<Line>:<Message>** |
---|
1599 | |
---|
1600 | A description of the fields follows: |
---|
1601 | |
---|
1602 | <Step> <Processor> <Task> <File> <Line> |
---|
1603 | See above **P** line. |
---|
1604 | |
---|
1605 | <Message> |
---|
1606 | An arbitrary message string. May contain colon characters (`:`). |
---|
1607 | |
---|
1608 | L |
---|
1609 | A log message line. It has the format: |
---|
1610 | |
---|
1611 | **L:<Message>** |
---|
1612 | |
---|
1613 | A description of the field follows: |
---|
1614 | |
---|
1615 | <Message> |
---|
1616 | An arbitrary message string. May contain colon characters (`:`). |
---|
1617 | |
---|
1618 | E |
---|
1619 | A test case end line. It has the format: |
---|
1620 | |
---|
1621 | **E:<TestCase>:N:<Steps>:F:<Failures>:D:<Duration>** |
---|
1622 | |
---|
1623 | A description of the fields follows: |
---|
1624 | |
---|
1625 | <TestCase> |
---|
1626 | A test case name. Must not contain colon characters (`:`). |
---|
1627 | |
---|
1628 | <Steps> |
---|
1629 | The final test step counter of a test case. Quiet test checks produce |
---|
1630 | no test steps. |
---|
1631 | |
---|
1632 | <Failures> |
---|
1633 | The count of failed test checks of a test case. |
---|
1634 | |
---|
1635 | <Duration> |
---|
1636 | The test case duration in seconds. |
---|
1637 | |
---|
1638 | Z |
---|
1639 | A test suite end line. It has the format: |
---|
1640 | |
---|
1641 | **Z:<TestSuite>:C:<TestCases>:N:<OverallSteps>:F:<OverallFailures>:D:<Duration>** |
---|
1642 | |
---|
1643 | A description of the fields follows: |
---|
1644 | |
---|
1645 | <TestSuite> |
---|
1646 | The test suite name. Must not contain colon characters (`:`). |
---|
1647 | |
---|
1648 | <TestCases> |
---|
1649 | The count of test cases in the test suite. |
---|
1650 | |
---|
1651 | <OverallSteps> |
---|
1652 | The overall count of test steps in the test suite. |
---|
1653 | |
---|
1654 | <OverallFailures> |
---|
1655 | The overall count of failed test cases in the test suite. |
---|
1656 | |
---|
1657 | <Duration> |
---|
1658 | The test suite duration in seconds. |
---|
1659 | |
---|
1660 | Y |
---|
1661 | Auxiliary information line. Issued after the test suite end. It has the format: |
---|
1662 | |
---|
1663 | **Y:ReportHash:SHA256:<Hash>** |
---|
1664 | |
---|
1665 | A description of the fields follows: |
---|
1666 | |
---|
1667 | <Hash> |
---|
1668 | The SHA256 hash value of the test suite report from the begin to the |
---|
1669 | end of the test suite. |
---|
1670 | |
---|
1671 | M |
---|
1672 | A code runtime measurement line. It has the formats: |
---|
1673 | |
---|
1674 | **M:B:<Name>** |
---|
1675 | |
---|
1676 | **M:V:<Variant>** |
---|
1677 | |
---|
1678 | **M:L:<Load>** |
---|
1679 | |
---|
1680 | **M:N:<SampleCount>** |
---|
1681 | |
---|
1682 | **M:S:<Count>:<Value>** |
---|
1683 | |
---|
1684 | **M:MI:<Minimum>** |
---|
1685 | |
---|
1686 | **M:Q1:<FirstQuartile>** |
---|
1687 | |
---|
1688 | **M:Q2:<Median>** |
---|
1689 | |
---|
1690 | **M:Q3:<ThirdQuartile>** |
---|
1691 | |
---|
1692 | **M:MX:<Maximum>** |
---|
1693 | |
---|
1694 | **M:MAD:<MedianAbsoluteDeviation>** |
---|
1695 | |
---|
1696 | **M:D:<SumOfSampleValues>** |
---|
1697 | |
---|
1698 | **M:E:<Name>:D:<Duration>** |
---|
1699 | |
---|
1700 | A description of the fields follows: |
---|
1701 | |
---|
1702 | <Name> |
---|
1703 | A code runtime measurement name. Must not contain colon characters |
---|
1704 | (`:`). |
---|
1705 | |
---|
1706 | <Variant> |
---|
1707 | The execution variant which is one of **ValidCache**, **HotCache**, |
---|
1708 | **DirtyCache**, or **Load**. |
---|
1709 | |
---|
1710 | <Load> |
---|
1711 | The active load workers count which ranges from one to the processor |
---|
1712 | count. |
---|
1713 | |
---|
1714 | <SampleCount> |
---|
1715 | The sample count as defined by the runtime measurement configuration. |
---|
1716 | |
---|
1717 | <Count> |
---|
1718 | The count of samples with the same value. |
---|
1719 | |
---|
1720 | <Value> |
---|
1721 | A sample value in seconds. |
---|
1722 | |
---|
1723 | <Minimum> |
---|
1724 | The minimum of the sample set in seconds. |
---|
1725 | |
---|
1726 | <FirstQuartile> |
---|
1727 | The first quartile of the sample set in seconds. |
---|
1728 | |
---|
1729 | <Median> |
---|
1730 | The median of the sample set in seconds. |
---|
1731 | |
---|
1732 | <ThirdQuartile> |
---|
1733 | The third quartile of the sample set in seconds. |
---|
1734 | |
---|
1735 | <Maximum> |
---|
1736 | The maximum of the sample set in seconds. |
---|
1737 | |
---|
1738 | <MedianAbsoluteDeviation> |
---|
1739 | The median absolute deviation of the sample set in seconds. |
---|
1740 | |
---|
1741 | <SumOfSampleValues> |
---|
1742 | The sum of all sample values of the sample set in seconds. |
---|
1743 | |
---|
1744 | <Duration> |
---|
1745 | The runtime measurement duration in seconds. It includes time to set |
---|
1746 | up the execution environment variant. |
---|
1747 | |
---|
1748 | .. code-block:: none |
---|
1749 | :caption: Example Test Report |
---|
1750 | |
---|
1751 | A:xyz |
---|
1752 | S:Platform:RTEMS |
---|
1753 | S:Compiler:7.4.0 20181206 (RTEMS 5, RSB e0aec65182449a4e22b820e773087636edaf5b32, Newlib 1d35a003f) |
---|
1754 | S:Version:5.0.0.820977c5af17c1ca2f79800d64bd87ce70a24c68 |
---|
1755 | S:BSP:erc32 |
---|
1756 | S:RTEMS_DEBUG:1 |
---|
1757 | S:RTEMS_MULTIPROCESSING:0 |
---|
1758 | S:RTEMS_POSIX_API:1 |
---|
1759 | S:RTEMS_PROFILING:0 |
---|
1760 | S:RTEMS_SMP:1 |
---|
1761 | B:timer |
---|
1762 | P:0:0:UI1:test-rtems.c:26 |
---|
1763 | P:1:0:UI1:test-rtems.c:29 |
---|
1764 | P:2:0:UI1:test-rtems.c:33 |
---|
1765 | P:3:0:ISR:test-rtems.c:14 |
---|
1766 | P:4:0:ISR:test-rtems.c:15 |
---|
1767 | P:5:0:UI1:test-rtems.c:38 |
---|
1768 | P:6:0:UI1:test-rtems.c:39 |
---|
1769 | P:7:0:UI1:test-rtems.c:42 |
---|
1770 | E:timer:N:8:F:0:D:0.019373 |
---|
1771 | B:rsc_success |
---|
1772 | P:0:0:UI1:test-rtems.c:59 |
---|
1773 | F:1:0:UI1:test-rtems.c:60:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL |
---|
1774 | F:*:0:UI1:test-rtems.c:62:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL |
---|
1775 | P:2:0:UI1:test-rtems.c:63 |
---|
1776 | F:3:0:UI1:test-rtems.c:64:RTEMS_INVALID_NUMBER == RTEMS_SUCCESSFUL |
---|
1777 | E:rsc_success:N:4:F:3:D:0.011128 |
---|
1778 | B:rsc |
---|
1779 | P:0:0:UI1:test-rtems.c:48 |
---|
1780 | F:1:0:UI1:test-rtems.c:49:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID |
---|
1781 | F:*:0:UI1:test-rtems.c:51:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID |
---|
1782 | P:2:0:UI1:test-rtems.c:52 |
---|
1783 | F:3:0:UI1:test-rtems.c:53:RTEMS_INVALID_NUMBER == RTEMS_INVALID_ID |
---|
1784 | E:rsc:N:4:F:3:D:0.011083 |
---|
1785 | Z:xyz:C:3:N:16:F:6:D:0.047201 |
---|
1786 | Y:ReportHash:SHA256:e5857c520dd9c9b7c15d4a76d78c21ccc46619c30a869ecd11bbcd1885155e0b |
---|
1787 | |
---|
1788 | Test Report Validation |
---|
1789 | ---------------------- |
---|
1790 | |
---|
1791 | You can add the `T_report_hash_sha256()` test suite action to the test suite |
---|
1792 | configuration to generate and report the SHA256 hash value of the test suite |
---|
1793 | report. The hash value covers everything reported by the test suite run from |
---|
1794 | the begin to the end. This can be used to check that the report generated on |
---|
1795 | the target is identical to the report received on the report consumer side. |
---|
1796 | The hash value is reported after the end of test suite line (`Z`) as auxiliary |
---|
1797 | information in a `Y` line. Consumers may have to reverse a `\\n` to `\\r\\n` |
---|
1798 | conversion before the hash is calculated. Such a conversion could be performed |
---|
1799 | by a particular put character handler provided by the test suite configuration. |
---|
1800 | |
---|
1801 | Supported Platforms |
---|
1802 | ------------------- |
---|
1803 | |
---|
1804 | The framework runs on FreeBSD, MSYS2, Linux and RTEMS. |
---|
1805 | |
---|
1806 | Test Framework Requirements for RTEMS |
---|
1807 | ===================================== |
---|
1808 | |
---|
1809 | The requirements on a test framework suitable for RTEMS are: |
---|
1810 | |
---|
1811 | License Requirements |
---|
1812 | -------------------- |
---|
1813 | |
---|
1814 | TF.License.Permissive |
---|
1815 | The test framework shall have a permissive open source license such as |
---|
1816 | BSD-2-Clause. |
---|
1817 | |
---|
1818 | Portability Requirements |
---|
1819 | ------------------------ |
---|
1820 | |
---|
1821 | TF.Portability |
---|
1822 | The test framework shall be portable. |
---|
1823 | |
---|
1824 | TF.Portability.RTEMS |
---|
1825 | The test framework shall run on RTEMS. |
---|
1826 | |
---|
1827 | TF.Portability.POSIX |
---|
1828 | The test framework shall be portable to POSIX compatible operating |
---|
1829 | systems. This allows to run test cases of standard C/POSIX/etc. APIs |
---|
1830 | on multiple platforms. |
---|
1831 | |
---|
1832 | TF.Portability.POSIX.Linux |
---|
1833 | The test framework shall run on Linux. |
---|
1834 | |
---|
1835 | TF.Portability.POSIX.FreeBSD |
---|
1836 | The test framework shall run on FreeBSD. |
---|
1837 | |
---|
1838 | TF.Portability.C11 |
---|
1839 | The test framework shall be written in C11. |
---|
1840 | |
---|
1841 | TF.Portability.Static |
---|
1842 | Test framework shall not use dynamic memory for basic services. |
---|
1843 | |
---|
1844 | TF.Portability.Small |
---|
1845 | The test framework shall be small enough to support low-end platforms |
---|
1846 | (e.g. 64KiB of RAM/ROM should be sufficient to test the architecture |
---|
1847 | port, e.g. no complex stuff such as file systems, etc.). |
---|
1848 | |
---|
1849 | TF.Portability.Small.LinkTimeConfiguration |
---|
1850 | The test framework shall be configured at link-time. |
---|
1851 | |
---|
1852 | TF.Portability.Small.Modular |
---|
1853 | The test framework shall be modular so that only necessary parts end up |
---|
1854 | in the final executable. |
---|
1855 | |
---|
1856 | TF.Portability.Small.Memory |
---|
1857 | The test framework shall not aggregate data during test case executions. |
---|
1858 | |
---|
1859 | Reporting Requirements |
---|
1860 | ---------------------- |
---|
1861 | |
---|
1862 | TF.Reporting |
---|
1863 | Test results shall be reported. |
---|
1864 | |
---|
1865 | TF.Reporting.Verbosity |
---|
1866 | The test report verbosity shall be configurable. This allows different |
---|
1867 | test run scenarios, e.g. regression test runs, full test runs with test |
---|
1868 | report verification against the planned test output. |
---|
1869 | |
---|
1870 | TF.Reporting.Verification |
---|
1871 | It shall be possible to use regular expressions to verify test reports |
---|
1872 | line by line. |
---|
1873 | |
---|
1874 | TF.Reporting.Compact |
---|
1875 | Test output shall be compact to avoid long test runs on platforms with |
---|
1876 | a slow output device, e.g. 9600 Baud UART. |
---|
1877 | |
---|
1878 | TF.Reporting.PutChar |
---|
1879 | A simple output one character function provided by the platform shall be |
---|
1880 | sufficient to report the test results. |
---|
1881 | |
---|
1882 | TF.Reporting.NonBlocking |
---|
1883 | The ouptut functions shall be non-blocking. |
---|
1884 | |
---|
1885 | TF.Reporting.Printf |
---|
1886 | The test framework shall provide printf()-like output functions. |
---|
1887 | |
---|
1888 | TF.Reporting.Printf.WithFP |
---|
1889 | There shall be a printf()-like output function with floating point |
---|
1890 | support. |
---|
1891 | |
---|
1892 | TF.Reporting.Printf.WithoutFP |
---|
1893 | There shall be a printf()-like output function without floating |
---|
1894 | point support on RTEMS. |
---|
1895 | |
---|
1896 | TF.Reporting.Platform |
---|
1897 | The test platform shall be reported. |
---|
1898 | |
---|
1899 | TF.Reporting.Platform.RTEMS.Git |
---|
1900 | The RTEMS source Git commit shall be reported. |
---|
1901 | |
---|
1902 | TF.Reporting.Platform.RTEMS.Arch |
---|
1903 | The RTEMS architecture name shall be reported. |
---|
1904 | |
---|
1905 | TF.Reporting.Platform.RTEMS.BSP |
---|
1906 | The RTEMS BSP name shall be reported. |
---|
1907 | |
---|
1908 | TF.Reporting.Platform.RTEMS.Tools |
---|
1909 | The RTEMS tool chain version shall be reported. |
---|
1910 | |
---|
1911 | TF.Reporting.Platform.RTEMS.Config.Debug |
---|
1912 | The shall be reported if RTEMS_DEBUG is defined. |
---|
1913 | |
---|
1914 | TF.Reporting.Platform.RTEMS.Config.Multiprocessing |
---|
1915 | The shall be reported if RTEMS_MULTIPROCESSING is defined. |
---|
1916 | |
---|
1917 | TF.Reporting.Platform.RTEMS.Config.POSIX |
---|
1918 | The shall be reported if RTEMS_POSIX_API is defined. |
---|
1919 | |
---|
1920 | TF.Reporting.Platform.RTEMS.Config.Profiling |
---|
1921 | The shall be reported if RTEMS_PROFILING is defined. |
---|
1922 | |
---|
1923 | TF.Reporting.Platform.RTEMS.Config.SMP |
---|
1924 | The shall be reported if RTEMS_SMP is defined. |
---|
1925 | |
---|
1926 | TF.Reporting.TestCase |
---|
1927 | The test cases shall be reported. |
---|
1928 | |
---|
1929 | TF.Reporting.TestCase.Begin |
---|
1930 | The test case begin shall be reported. |
---|
1931 | |
---|
1932 | TF.Reporting.TestCase.End |
---|
1933 | The test case end shall be reported. |
---|
1934 | |
---|
1935 | TF.Reporting.TestCase.Tests |
---|
1936 | The count of test checks of the test case shall be reported. |
---|
1937 | |
---|
1938 | TF.Reporting.TestCase.Failures |
---|
1939 | The count of failed test checks of the test case shall be reported. |
---|
1940 | |
---|
1941 | TF.Reporting.TestCase.Timing |
---|
1942 | Test case timing shall be reported. |
---|
1943 | |
---|
1944 | TF.Reporting.TestCase.Tracing |
---|
1945 | Automatic tracing and reporting of thread context switches and |
---|
1946 | interrupt service routines shall be optionally performed. |
---|
1947 | |
---|
1948 | Environment Requirements |
---|
1949 | ------------------------ |
---|
1950 | |
---|
1951 | TF.Environment |
---|
1952 | The test framework shall support all environment conditions of the platform. |
---|
1953 | |
---|
1954 | TF.Environment.SystemStart |
---|
1955 | The test framework shall run during early stages of the system start, |
---|
1956 | e.g. valid stack pointer, initialized data and cleared BSS, nothing |
---|
1957 | more. |
---|
1958 | |
---|
1959 | TF.Environment.BeforeDeviceDrivers |
---|
1960 | The test framework shall run before device drivers are initialized. |
---|
1961 | |
---|
1962 | TF.Environment.InterruptContext |
---|
1963 | The test framework shall support test case code in interrupt context. |
---|
1964 | |
---|
1965 | Usability Requirements |
---|
1966 | ---------------------- |
---|
1967 | |
---|
1968 | TF.Usability |
---|
1969 | The test framework shall be easy to use. |
---|
1970 | |
---|
1971 | TF.Usability.TestCase |
---|
1972 | It shall be possible to write test cases. |
---|
1973 | |
---|
1974 | TF.Usability.TestCase.Independence |
---|
1975 | It shall be possible to write test cases in modules independent of |
---|
1976 | the test runner. |
---|
1977 | |
---|
1978 | TF.Usability.TestCase.AutomaticRegistration |
---|
1979 | Test cases shall be registered automatically, e.g. via constructors |
---|
1980 | or linker sets. |
---|
1981 | |
---|
1982 | TF.Usability.TestCase.Order |
---|
1983 | It shall be possible to sort the registered test cases (e.g. random, |
---|
1984 | by name) before they are executed. |
---|
1985 | |
---|
1986 | TF.Usability.TestCase.Resources |
---|
1987 | It shall be possible to use resources with a life time restricted to |
---|
1988 | the test case. |
---|
1989 | |
---|
1990 | TF.Usability.TestCase.Resources.Memory |
---|
1991 | It shall be possible to dynamically allocate memory which is |
---|
1992 | automatically freed once the test case completed. |
---|
1993 | |
---|
1994 | TF.Usability.TestCase.Resources.File |
---|
1995 | It shall be possible to create a file which is automatically |
---|
1996 | unlinked once the test case completed. |
---|
1997 | |
---|
1998 | TF.Usability.TestCase.Resources.Directory |
---|
1999 | It shall be possible to create a directory which is automatically |
---|
2000 | removed once the test case completed. |
---|
2001 | |
---|
2002 | TF.Usability.TestCase.Resources.FileDescriptor |
---|
2003 | It shall be possible to open a file descriptor which is |
---|
2004 | automatically closed once the test case completed. |
---|
2005 | |
---|
2006 | TF.Usability.TestCase.Fixture |
---|
2007 | It shall be possible to use a text fixture for test cases. |
---|
2008 | |
---|
2009 | TF.Usability.TestCase.Fixture.SetUp |
---|
2010 | It shall be possible to provide a set up handler for each test case. |
---|
2011 | |
---|
2012 | TF.Usability.TestCase.Fixture.TearDown |
---|
2013 | It shall be possible to provide a tear down handler for each test |
---|
2014 | case. |
---|
2015 | |
---|
2016 | TF.Usability.TestCase.Context |
---|
2017 | The test case context shall be verified a certain points. |
---|
2018 | |
---|
2019 | TF.Usability.TestCase.Context.VerifyAtEnd |
---|
2020 | After a test case exection it shall be verified that the context |
---|
2021 | is equal to the context at the test case begin. This helps to |
---|
2022 | ensure that test cases are independent of each other. |
---|
2023 | |
---|
2024 | TF.Usability.TestCase.Context.VerifyThread |
---|
2025 | The test framework shall provide a function to ensure that the |
---|
2026 | test case code executes in normal thread context. This helps |
---|
2027 | to ensure that operating system service calls return to a sane |
---|
2028 | context. |
---|
2029 | |
---|
2030 | TF.Usability.TestCase.Context.Configurable |
---|
2031 | The context verified in test case shall be configurable at link-time. |
---|
2032 | |
---|
2033 | TF.Usability.TestCase.Context.ThreadDispatchDisableLevel |
---|
2034 | It shall be possible to verify the thread dispatch disable level. |
---|
2035 | |
---|
2036 | TF.Usability.TestCase.Context.ISRNestLevel |
---|
2037 | It shall be possible to verify the ISR nest level. |
---|
2038 | |
---|
2039 | TF.Usability.TestCase.Context.InterruptLevel |
---|
2040 | It shall be possible to verify the interrupt level (interrupts |
---|
2041 | enabled/disabled). |
---|
2042 | |
---|
2043 | TF.Usability.TestCase.Context.Workspace |
---|
2044 | It shall be possible to verify the workspace. |
---|
2045 | |
---|
2046 | TF.Usability.TestCase.Context.Heap |
---|
2047 | It shall be possible to verify the heap. |
---|
2048 | |
---|
2049 | TF.Usability.TestCase.Context.OpenFileDescriptors |
---|
2050 | It shall be possible to verify the open file descriptors. |
---|
2051 | |
---|
2052 | TF.Usability.TestCase.Context.Classic |
---|
2053 | It shall be possible to verify Classic API objects. |
---|
2054 | |
---|
2055 | TF.Usability.TestCase.Context.Classic.Barrier |
---|
2056 | It shall be possible to verify Classic API Barrier objects. |
---|
2057 | |
---|
2058 | TF.Usability.TestCase.Context.Classic.Extensions |
---|
2059 | It shall be possible to verify Classic API User Extensions |
---|
2060 | objects. |
---|
2061 | |
---|
2062 | TF.Usability.TestCase.Context.Classic.MessageQueues |
---|
2063 | It shall be possible to verify Classic API Message Queue |
---|
2064 | objects. |
---|
2065 | |
---|
2066 | TF.Usability.TestCase.Context.Classic.Partitions |
---|
2067 | It shall be possible to verify Classic API Partition objects. |
---|
2068 | |
---|
2069 | TF.Usability.TestCase.Context.Classic.Periods |
---|
2070 | It shall be possible to verify Classic API Rate Monotonic |
---|
2071 | Period objects. |
---|
2072 | |
---|
2073 | TF.Usability.TestCase.Context.Classic.Regions |
---|
2074 | It shall be possible to verify Classic API Region objects. |
---|
2075 | |
---|
2076 | TF.Usability.TestCase.Context.Classic.Semaphores |
---|
2077 | It shall be possible to verify Classic API Semaphore |
---|
2078 | objects. |
---|
2079 | |
---|
2080 | TF.Usability.TestCase.Context.Classic.Tasks |
---|
2081 | It shall be possible to verify Classic API Task objects. |
---|
2082 | |
---|
2083 | TF.Usability.TestCase.Context.Classic.Timers |
---|
2084 | It shall be possible to verify Classic API Timer objects. |
---|
2085 | |
---|
2086 | TF.Usability.TestCase.Context.POSIX |
---|
2087 | It shall be possible to verify POSIX API objects. |
---|
2088 | |
---|
2089 | TF.Usability.TestCase.Context.POSIX.Keys |
---|
2090 | It shall be possible to verify POSIX API Key objects. |
---|
2091 | |
---|
2092 | TF.Usability.TestCase.Context.POSIX.KeyValuePairs |
---|
2093 | It shall be possible to verify POSIX API Key Value Pair |
---|
2094 | objects. |
---|
2095 | |
---|
2096 | TF.Usability.TestCase.Context.POSIX.MessageQueues |
---|
2097 | It shall be possible to verify POSIX API Message Queue |
---|
2098 | objects. |
---|
2099 | |
---|
2100 | TF.Usability.TestCase.Context.POSIX.Semaphores |
---|
2101 | It shall be possible to verify POSIX API Named Semaphores |
---|
2102 | objects. |
---|
2103 | |
---|
2104 | TF.Usability.TestCase.Context.POSIX.Shms |
---|
2105 | It shall be possible to verify POSIX API Shared Memory |
---|
2106 | objects. |
---|
2107 | |
---|
2108 | TF.Usability.TestCase.Context.POSIX.Threads |
---|
2109 | It shall be possible to verify POSIX API Thread objects. |
---|
2110 | |
---|
2111 | TF.Usability.TestCase.Context.POSIX.Timers |
---|
2112 | It shall be possible to verify POSIX API Timer objects. |
---|
2113 | |
---|
2114 | TF.Usability.Assert |
---|
2115 | There shall be functions to assert test objectives. |
---|
2116 | |
---|
2117 | TF.Usability.Assert.Safe |
---|
2118 | Test assert functions shall be safe to use, e.g. assert(a == b) vs. |
---|
2119 | assert(a = b) vs. assert_eq(a, b). |
---|
2120 | |
---|
2121 | TF.Usability.Assert.Continue |
---|
2122 | There shall be assert functions which allow the test case to |
---|
2123 | continue in case of an assertion failure. |
---|
2124 | |
---|
2125 | TF.Usability.Assert.Abort |
---|
2126 | There shall be assert functions which abourt the test case in case |
---|
2127 | of an assertion failure. |
---|
2128 | |
---|
2129 | TF.Usability.EasyToWrite |
---|
2130 | It shall be easy to write test code, e.g. avoid long namespace prefix |
---|
2131 | rtems_test_*. |
---|
2132 | |
---|
2133 | TF.Usability.Threads |
---|
2134 | The test framework shall support multi-threading. |
---|
2135 | |
---|
2136 | TF.Usability.Pattern |
---|
2137 | The test framework shall support test patterns. |
---|
2138 | |
---|
2139 | TF.Usability.Pattern.Interrupts |
---|
2140 | The test framework shall support test cases which use interrupts, |
---|
2141 | e.g. spintrcritical*. |
---|
2142 | |
---|
2143 | TF.Usability.Pattern.Parallel |
---|
2144 | The test framework shall support test cases which want to run code |
---|
2145 | in parallel on SMP machines. |
---|
2146 | |
---|
2147 | TF.Usability.Pattern.Timing |
---|
2148 | The test framework shall support test cases which want to measure |
---|
2149 | the timing of code sections under various platform conditions, e.g. |
---|
2150 | dirty cache, empty cache, hot cache, with load from other |
---|
2151 | processors, etc.. |
---|
2152 | |
---|
2153 | TF.Usability.Configuration |
---|
2154 | The test framework shall be configurable. |
---|
2155 | |
---|
2156 | TF.Usability.Configuration.Time |
---|
2157 | The timestamp function shall be configurable, e.g. to allow test |
---|
2158 | runs without a clock driver. |
---|
2159 | |
---|
2160 | Performance Requirements |
---|
2161 | ------------------------ |
---|
2162 | |
---|
2163 | TF.Performance.RTEMS.No64BitDivision |
---|
2164 | The test framework shall not use 64-bit divisions on RTEMS. |
---|
2165 | |
---|
2166 | Off-the-shelf Test Frameworks |
---|
2167 | ============================= |
---|
2168 | |
---|
2169 | There are several |
---|
2170 | `off-the-shelf test frameworks for C/C++ <https://en.wikipedia.org/wiki/List_of_unit_testing_frameworks#C>`_. |
---|
2171 | The first obstacle for test frameworks is the license requirement |
---|
2172 | (`TF.License.Permissive`). |
---|
2173 | |
---|
2174 | bdd-for-c |
---|
2175 | --------- |
---|
2176 | |
---|
2177 | In the `bdd-for-c <https://github.com/grassator/bdd-for-c>`_ framework the |
---|
2178 | complete test suite must be contained in one file and the main function is |
---|
2179 | generated. This violates `TF.Usability.TestCase.Independence`. |
---|
2180 | |
---|
2181 | CBDD |
---|
2182 | ---- |
---|
2183 | |
---|
2184 | The `CBDD <https://github.com/nassersala/cbdd>`_ framework uses the |
---|
2185 | `C blocks <https://clang.llvm.org/docs/BlockLanguageSpec.html>`_ extension from |
---|
2186 | clang. This violates `TF.Portability.C11`. |
---|
2187 | |
---|
2188 | Google Test |
---|
2189 | ----------- |
---|
2190 | |
---|
2191 | `Google Test 1.8.1 <https://git.rtems.org/sebh/rtems-gtest.git/>`_ |
---|
2192 | is supported by RTEMS. Unfortunately, it is written in C++ and is to heavy |
---|
2193 | weight for low-end platforms. Otherwise it is a nice framework. |
---|
2194 | |
---|
2195 | Unity |
---|
2196 | ----- |
---|
2197 | |
---|
2198 | The `Unity Test API <https://github.com/ThrowTheSwitch/Unity>`_ does not meet |
---|
2199 | our requirements. There was a `discussion on the mailing list in 2013 |
---|
2200 | <https://lists.rtems.org/pipermail/devel/2013-September/004499.html>`_. |
---|
2201 | |
---|
2202 | Standard Test Report Formats |
---|
2203 | ============================ |
---|
2204 | |
---|
2205 | JUnit XML |
---|
2206 | --------- |
---|
2207 | |
---|
2208 | A common test report format is `JUnit XML <http://llg.cubic.org/docs/junit/>`_. |
---|
2209 | |
---|
2210 | .. code-block:: xml |
---|
2211 | |
---|
2212 | <?xml version="1.0" encoding="UTF-8" ?> |
---|
2213 | <testsuites id="xyz" name="abc" tests="225" failures="1262" time="0.001"> |
---|
2214 | <testsuite id="def" name="ghi" tests="45" failures="17" time="0.001"> |
---|
2215 | <testcase id="jkl" name="mno" time="0.001"> |
---|
2216 | <failure message="pqr" type="stu"></failure> |
---|
2217 | <system-out>stdout</system-out> |
---|
2218 | <system-err>stderr</system-err> |
---|
2219 | </testcase> |
---|
2220 | </testsuite> |
---|
2221 | </testsuites> |
---|
2222 | |
---|
2223 | The major problem with this format is that you have to output the failure count |
---|
2224 | of all test suites and the individual test suite before the test case output. |
---|
2225 | You know the failure count only after a complete test run. This runs contrary |
---|
2226 | to requirement `TF.Portability.Small.Memory`. It is also a bit verbose |
---|
2227 | (`TF.Reporting.Compact`). |
---|
2228 | |
---|
2229 | It is easy to convert a full test report generated by :ref:`The RTEMS Test |
---|
2230 | Framework <RTEMSTestFramework>` to the JUnit XML format. |
---|
2231 | |
---|
2232 | Test Anything Protocol |
---|
2233 | ---------------------- |
---|
2234 | |
---|
2235 | The |
---|
2236 | `Test Anything Protocol <http://testanything.org/>`_ |
---|
2237 | (TAP) is easy to consume and produce. |
---|
2238 | |
---|
2239 | .. code-block:: none |
---|
2240 | |
---|
2241 | 1..4 |
---|
2242 | ok 1 - Input file opened |
---|
2243 | not ok 2 - First line of the input valid |
---|
2244 | ok 3 - Read the rest of the file |
---|
2245 | not ok 4 - Summarized correctly # TODO Not written yet |
---|
2246 | |
---|
2247 | You have to know in advance how many test statements you want to execute in a |
---|
2248 | test case. The problem with this format is that there is no standard way to |
---|
2249 | provide auxiliary data such as test timing or a tracing report. |
---|
2250 | |
---|
2251 | It is easy to convert a full test report generated by :ref:`The RTEMS Test |
---|
2252 | Framework <RTEMSTestFramework>` to the TAP format. |
---|