1 | .. comment SPDX-License-Identifier: CC-BY-SA-4.0 |
---|
2 | |
---|
3 | .. Copyright (C) 2017 Chris Johns <chrisj@rtems.org> |
---|
4 | |
---|
5 | .. _rtems-tester-command: |
---|
6 | |
---|
7 | RTEMS Tester and Run |
---|
8 | ==================== |
---|
9 | |
---|
10 | .. index:: Tools, rtems-test, rtems-run |
---|
11 | |
---|
12 | The RTEMS Tester is a test tool that provides a command line interface to run |
---|
13 | test executable on supported targets. The tool provides back-end support for |
---|
14 | common simulators, debuggers and boot loaders. Board support package (BSP) |
---|
15 | configurations for RTEMS are provided and can be used to run all the tests in |
---|
16 | the RTEMS test suite. The tool and it's framework is not specific to RTEMS and |
---|
17 | can be configured to run any suitable application. |
---|
18 | |
---|
19 | RTEMS is an embedded operating system and is cross-compiled on a range of host |
---|
20 | machines. The executables run on the target hardware and this can vary widely |
---|
21 | from open source simulators, commercial simulators, debuggers with simulators, |
---|
22 | debuggers with hardware specific pods and devices to targe boot |
---|
23 | loaders. Testing RTEMS requires the cross-compiled test executable is |
---|
24 | transferred to the target hardware, executed and the output captured and |
---|
25 | returned to the test host where it is analyzed to determine the test |
---|
26 | result. |
---|
27 | |
---|
28 | Running the RTEMS tests on your target is very important. It provides you with |
---|
29 | a traceable record showing that your RTEMS version and its tools are working at |
---|
30 | the level the RTEMS development team expect when releasing RTEMS. Being able to |
---|
31 | easily run the tests and verify the results is critical in maintaining a high |
---|
32 | standard. |
---|
33 | |
---|
34 | Available BSP testers |
---|
35 | --------------------- |
---|
36 | |
---|
37 | You can list the available BSP testers with: |
---|
38 | |
---|
39 | .. code-block:: shell |
---|
40 | |
---|
41 | $ rtems-test --list-bsps |
---|
42 | arm920 |
---|
43 | beagleboardxm |
---|
44 | beagleboneblack |
---|
45 | jmr3904-run |
---|
46 | jmr3904 |
---|
47 | mcf5235 |
---|
48 | pc |
---|
49 | psim-run |
---|
50 | psim |
---|
51 | realview_pbx_a9_qemu |
---|
52 | sis-run |
---|
53 | sis |
---|
54 | xilinx_zynq_a9_qemu |
---|
55 | xilinx_zynq_a9_qemu_smp |
---|
56 | xilinx_zynq_zc706 |
---|
57 | xilinx_zynq_zc706_qemu |
---|
58 | xilinx_zynq_zedboard |
---|
59 | |
---|
60 | .. note:: The list is growing all the time and if your BSP is not supported we |
---|
61 | encourage you to add it and submit the configuration back to the |
---|
62 | project. |
---|
63 | |
---|
64 | Some of the BSPs may appear more than once in the list. These are aliased BSP |
---|
65 | configurations that may use a different back-end. An example is the erc32 BSP. |
---|
66 | There is the ``erc32`` tester which uses the GDB back-end and the |
---|
67 | ``erc32-run`` tester which uses the ``run`` command for erc32. We will show |
---|
68 | how to use :program:`rtems-test` command with the erc32 BSP because it is easy |
---|
69 | to build an use. |
---|
70 | |
---|
71 | Building RTEMS Tests |
---|
72 | -------------------- |
---|
73 | |
---|
74 | Build the RTEMS Kernel (See :ref:`rtems-kernel`) by cloning the repository, |
---|
75 | running the ``bootstrap`` procedure, building and finally installing the |
---|
76 | kernel. Be sure to enable tests by using ``--enable-tests`` option with |
---|
77 | configure after running ``bootstrap``. |
---|
78 | |
---|
79 | .. code-block:: shell |
---|
80 | |
---|
81 | $ ../../rtems.git/configure --target=sparc-rtems5 \ |
---|
82 | --enable-tests --enable-rtemsbsp=erc32 |
---|
83 | $ make |
---|
84 | |
---|
85 | Add the `-j` option to the make command with the number of cores to run a |
---|
86 | parallel build. |
---|
87 | |
---|
88 | Building all the tests takes time and it uses more disk so be patient. When |
---|
89 | finished all the tests will have been built. Some BSPs may require a post-build |
---|
90 | process to be run on the RTEMS ELF executable to create an image suitable for |
---|
91 | execution. This can be built into the configuration script and the tester will |
---|
92 | perform a pre-test command to covert the executable to a suitable format for |
---|
93 | your target. |
---|
94 | |
---|
95 | Before running all the tests it is a good idea to run the ``hello`` test. The |
---|
96 | ``hello`` test is an RTEMS version of the classic "Hello World" example and |
---|
97 | running it shows you have a working tool chain and build of RTEMS ready to run |
---|
98 | the tests. Using the run with the ERC32 BSP the command is: |
---|
99 | |
---|
100 | .. code-block:: shell |
---|
101 | |
---|
102 | $ sparc-rtems5-run sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe |
---|
103 | |
---|
104 | *** BEGIN OF TEST HELLO WORLD *** |
---|
105 | Hello World |
---|
106 | *** END OF TEST HELLO WORLD *** |
---|
107 | |
---|
108 | The run command is the GDB simulator without the GDB part. |
---|
109 | |
---|
110 | Running the example using GDB: |
---|
111 | |
---|
112 | .. code-block:: shell |
---|
113 | |
---|
114 | $ sparc-rtems5-gdb sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe |
---|
115 | GNU gdb (GDB) 7.12 |
---|
116 | Copyright (C) 2016 Free Software Foundation, Inc. |
---|
117 | License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> |
---|
118 | This is free software: you are free to change and redistribute it. |
---|
119 | There is NO WARRANTY, to the extent permitted by law. Type "show copying" |
---|
120 | and "show warranty" for details. |
---|
121 | This GDB was configured as "--host=x86_64-linux-gnu --target=sparc-rtems5". |
---|
122 | Type "show configuration" for configuration details. |
---|
123 | For bug reporting instructions, please see: |
---|
124 | <http://www.gnu.org/software/gdb/bugs/>. |
---|
125 | Find the GDB manual and other documentation resources online at: |
---|
126 | <http://www.gnu.org/software/gdb/documentation/>. |
---|
127 | For help, type "help". |
---|
128 | Type "apropos word" to search for commands related to "word"... |
---|
129 | Reading symbols from |
---|
130 | sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe...done. |
---|
131 | (gdb) target sim |
---|
132 | Connected to the simulator. |
---|
133 | (gdb) load |
---|
134 | (gdb) r |
---|
135 | Starting program: sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe |
---|
136 | |
---|
137 | |
---|
138 | *** BEGIN OF TEST HELLO WORLD *** |
---|
139 | Hello World |
---|
140 | *** END OF TEST HELLO WORLD *** |
---|
141 | [Inferior 1 (process 42000) exited normally] |
---|
142 | (gdb) q |
---|
143 | |
---|
144 | The command ``r`` is used to debug set break points before issuing the GDB |
---|
145 | ``run`` command. |
---|
146 | |
---|
147 | There are currently close to 500 separate tests and you can run them all with a |
---|
148 | single RTEMS Tester command. |
---|
149 | |
---|
150 | Running the Tests |
---|
151 | ----------------- |
---|
152 | |
---|
153 | The :program:`rtems-test` command line accepts a range of options. These are |
---|
154 | discussed later in the manual. Any command line argument without a `--` prefix |
---|
155 | is a test executable. You can pass more than one executable on the command |
---|
156 | line. If the executable is a path to a directory the directories under that |
---|
157 | path are searched for any file with a ``.exe`` extension. This is the default |
---|
158 | extension for RTEMS executables built within RTEMS. |
---|
159 | |
---|
160 | To run the erc32 tests enter the following command from the top of the erc32 |
---|
161 | BSP build tree: |
---|
162 | |
---|
163 | .. code-block:: shell |
---|
164 | |
---|
165 | $ ~/development/rtems/test/rtems-tools.git/tester/rtems-test \ |
---|
166 | --log=log_erc32_run \ |
---|
167 | --rtems-bsp=erc32-run \ |
---|
168 | --rtems-tools=$HOME/development/rtems/5 \ |
---|
169 | sparc-rtems5/c/erc32/testsuites/samples |
---|
170 | RTEMS Testing - Tester, 5.not_released |
---|
171 | [ 1/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: base_sp.exe |
---|
172 | [ 2/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: capture.exe |
---|
173 | [ 3/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: cdtest.exe |
---|
174 | [ 4/13] p:0 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: fileio.exe |
---|
175 | [ 5/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: hello.exe |
---|
176 | [ 6/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: cxx_iostream.exe |
---|
177 | [ 8/13] p:2 f:0 u:0 e:0 I:0 B:0 t:2 i:0 | sparc/erc32: minimum.exe |
---|
178 | [ 7/13] p:2 f:0 u:0 e:0 I:0 B:0 t:2 i:0 | sparc/erc32: loopback.exe |
---|
179 | [ 9/13] p:3 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: nsecs.exe |
---|
180 | [10/13] p:3 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: paranoia.exe |
---|
181 | [11/13] p:4 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: pppd.exe |
---|
182 | [12/13] p:6 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: ticker.exe |
---|
183 | [13/13] p:6 f:0 u:0 e:0 I:0 B:0 t:3 i:0 | sparc/erc32: unlimited.exe |
---|
184 | Passed: 7 |
---|
185 | Failed: 0 |
---|
186 | User Input: 0 |
---|
187 | Expected Fail: 0 |
---|
188 | Indeterminate: 0 |
---|
189 | Benchmark: 0 |
---|
190 | Timeout: 5 |
---|
191 | Invalid: 1 |
---|
192 | Total: 13 |
---|
193 | Average test time: 0:00:27.963000 |
---|
194 | Testing time : 0:06:03.519012 |
---|
195 | |
---|
196 | * The RTEMS Tester's test command. In this example we are using an absolute |
---|
197 | path. |
---|
198 | * The ``--log`` option sends the output to a log file. By default only failed |
---|
199 | tests log the complete output. |
---|
200 | * Select the erc32 BSP and use GDB. |
---|
201 | * Path to the RTEMS tools so GDB can be found. |
---|
202 | * Path to the erc32 BSP built with all tests to run. If you add subdirectories |
---|
203 | to the path specific tests can be run. |
---|
204 | * The output has been shortened so it fits nicely here. |
---|
205 | * The test results shows passes, fails, timeouts, and invalid results. In |
---|
206 | this run 13 tests passed and 5 tests timed out and 1 is invalid. The |
---|
207 | timeouts are probably due to the tests not having enough execute time to |
---|
208 | complete. The default timeout is 180 seconds and some of the interrupt tests |
---|
209 | need longer. The amount of time depends on the performance of your host CPU |
---|
210 | running the simulations. |
---|
211 | * The output shows the average time per test and the total time taken to run |
---|
212 | all the tests. |
---|
213 | * If the path to the testsuites was put to |
---|
214 | ``sparc-rtems5/c/erc32/testsuites`` instead of |
---|
215 | ``sparc-rtems5/c/erc32/testsuites/samples`` then all the executables |
---|
216 | would have been tested and not just those in samples. |
---|
217 | |
---|
218 | This BSP requires the ``--rtems-tools`` option because the SPARC GDB is the |
---|
219 | ``sparc-rtems4.11-gdb`` command that is part of the RTEMS tools. Not every BSP |
---|
220 | will require this option so you will need to check the specifics of the BSP |
---|
221 | configuration to determine if it is needed. |
---|
222 | |
---|
223 | The output you see is each test starting to run. The :program:`rtems-test` |
---|
224 | command by default runs multiple tests in parallel so you will see a number |
---|
225 | start quickly and then new tests start as others finish. The output shown here |
---|
226 | is from an 8 core processor so the first 8 are started in parallel and the |
---|
227 | status shows the order in which they actually started, which is not 1 to 8. |
---|
228 | |
---|
229 | The test start line shows the current status of the tests. The status reported |
---|
230 | is when the test starts and not the result of that test. A fail, timeout or |
---|
231 | invalid count changing means a test running before this test started failed, |
---|
232 | not the starting test. The status here has 7 tests passed, no failures, 5 |
---|
233 | timeouts and 1 invalid test. |
---|
234 | |
---|
235 | .. code-block:: shell |
---|
236 | |
---|
237 | [ 5/13] p:2 f:0 u:0 e:0 I:0 B:0 t:0 i:0 | sparc/erc32: hello.exe |
---|
238 | |
---|
239 | * [ 5/13] indicates the test number, in this case test 5 of 13 tests. |
---|
240 | * ``p`` is the passed test count (2 in this case) |
---|
241 | * ``f`` is the failed test count (0 in this case) |
---|
242 | * ``u`` is the count for test marked as "user-input" as they expect input from |
---|
243 | user |
---|
244 | * ``e`` is the expected-fail count (tests that are expected to fail) |
---|
245 | * ``I`` is the count for tests the results of which are indeterminate |
---|
246 | * ``B`` is the count for benchmarked tests |
---|
247 | * ``t`` is the timeout test count |
---|
248 | * ``i`` is the invalid test count. |
---|
249 | * ``sparc/erc32`` is the architecture and BSP names. |
---|
250 | * ``hello.exe`` is the executable name. |
---|
251 | |
---|
252 | The test log records all the tests and results. The reporting mode by default |
---|
253 | only provides the output history if a test fails, times out, or is invalid. The |
---|
254 | time taken by each test is also recorded. |
---|
255 | |
---|
256 | The tests must complete in a specified time or the test is marked as timed |
---|
257 | out. The default timeout is 3 minutes and can be globally changed using the |
---|
258 | ``--timeout`` command line option. The time required to complete a test can |
---|
259 | vary. When simulators are run in parallel the time taken depends on the |
---|
260 | specifics of the host machine being used. A test per core is the most stable |
---|
261 | method even though more tests can be run than available cores. If your machine |
---|
262 | needs longer or you are using a VM you may need to lengthen the timeout. |
---|
263 | |
---|
264 | Test Status |
---|
265 | ----------- |
---|
266 | |
---|
267 | Tests can be marked with one of the following: |
---|
268 | |
---|
269 | * Pass |
---|
270 | * Fail |
---|
271 | * User-input |
---|
272 | * Expected-fail |
---|
273 | * Indeterminate |
---|
274 | * Benchmark |
---|
275 | * Timeout |
---|
276 | * Invalid |
---|
277 | |
---|
278 | The RTEMS console or ``stdout`` output from the test is needed to determine the |
---|
279 | result of the test. |
---|
280 | |
---|
281 | Pass |
---|
282 | ^^^^ |
---|
283 | A test passes if the start and end markers are seen in the test output. The |
---|
284 | start marker is ``***`` and the end mark is ``*** END OF TEST``. All tests in |
---|
285 | the RTEMS test suite have these markers. |
---|
286 | |
---|
287 | Fail |
---|
288 | ^^^^ |
---|
289 | A test fails if the start marker is seen and there is no end marker. |
---|
290 | |
---|
291 | User-input |
---|
292 | ^^^^^^^^^^ |
---|
293 | A test marked as "user-input" as it expects input from user |
---|
294 | |
---|
295 | Expected-fail |
---|
296 | ^^^^^^^^^^^^^ |
---|
297 | A test that is expected to fail. |
---|
298 | |
---|
299 | Indeterminate |
---|
300 | ^^^^^^^^^^^^^ |
---|
301 | A test the results of which are indeterminate. |
---|
302 | |
---|
303 | Benchmark |
---|
304 | ^^^^^^^^^ |
---|
305 | A benchmarked test. |
---|
306 | |
---|
307 | Timeout |
---|
308 | ^^^^^^^ |
---|
309 | If the test does not complete within the timeout setting the test is marked as |
---|
310 | having timed out. |
---|
311 | |
---|
312 | Invalid |
---|
313 | ^^^^^^^ |
---|
314 | If no start marker is seen the test is marked as invalid. If you are testing on |
---|
315 | real target hardware things can sometimes go wrong and the target may not |
---|
316 | initialize or respond to the debugger in an expected way. |
---|
317 | |
---|
318 | Reporting |
---|
319 | --------- |
---|
320 | |
---|
321 | The report written to the log has the following modes: |
---|
322 | |
---|
323 | * All (``all``) |
---|
324 | * Failures (``failures``) |
---|
325 | * None (``none``) |
---|
326 | |
---|
327 | The mode is controlled using the command line option ``--report-mode`` using |
---|
328 | the values listed above. |
---|
329 | |
---|
330 | All |
---|
331 | ^^^ |
---|
332 | The output of all tests is written to the log. |
---|
333 | |
---|
334 | Failures |
---|
335 | ^^^^^^^^ |
---|
336 | The output of the all tests that do not pass is written to the log. |
---|
337 | |
---|
338 | None |
---|
339 | ^^^^ |
---|
340 | No output is written to the log. |
---|
341 | |
---|
342 | The output is tagged so you can determine where it comes from. The following is |
---|
343 | the complete output for the In Memory File System test ``imfs_fslink.exe`` |
---|
344 | running on a Coldfire MCF5235 using GDB and a BDM pod: |
---|
345 | |
---|
346 | .. code-block:: shell |
---|
347 | |
---|
348 | [ 11/472] p:9 f:0 t:0 i:1 | m68k/mcf5235: imfs_fslink.exe |
---|
349 | > gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe |
---|
350 | > Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe... |
---|
351 | > done. |
---|
352 | > target remote | m68k-bdm-gdbserver pipe 003-005 |
---|
353 | > Remote debugging using | m68k-bdm-gdbserver pipe 003-005 |
---|
354 | > m68k-bdm: debug module version 0 |
---|
355 | > m68k-bdm: detected MCF5235 |
---|
356 | > m68k-bdm: architecture CF5235 connected to 003-005 |
---|
357 | > m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282) |
---|
358 | > Process 003-005 created; pid = 0 |
---|
359 | > 0x00006200 in ?? () |
---|
360 | > thb *0xffe254c0 |
---|
361 | > Hardware assisted breakpoint 1 at 0xffe254c0 |
---|
362 | > continue |
---|
363 | > Continuing. |
---|
364 | ] |
---|
365 | ] |
---|
366 | ] External Reset |
---|
367 | ] |
---|
368 | ] ColdFire MCF5235 on the BCC |
---|
369 | ] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28) |
---|
370 | ] Copyright 1995-2004 Freescale Semiconductor, Inc. All Rights Reserved. |
---|
371 | ] |
---|
372 | ] Enter 'help' for help. |
---|
373 | ] |
---|
374 | > Temporary breakpoint |
---|
375 | > 1, 0xffe254c0 in ?? () |
---|
376 | > load |
---|
377 | > Loading section .text, size 0x147e0 lma 0x40000 |
---|
378 | > Loading section .data, size 0x5d0 lma 0x547e0 |
---|
379 | > Start address 0x40414, load size 85424 |
---|
380 | > Transfer rate: 10 KB/sec, 1898 bytes/write. |
---|
381 | > b bsp_reset |
---|
382 | > Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14. |
---|
383 | > continue |
---|
384 | > Continuing. |
---|
385 | ] dBUG> |
---|
386 | ] |
---|
387 | ] *** FILE SYSTEM TEST ( IMFS ) *** |
---|
388 | ] Initializing filesystem IMFS |
---|
389 | ] |
---|
390 | ] |
---|
391 | ] *** LINK TEST *** |
---|
392 | ] link creates hardlinks |
---|
393 | ] test if the stat is the same |
---|
394 | ] chmod and chown |
---|
395 | ] unlink then stat the file |
---|
396 | ] *** END OF LINK TEST *** |
---|
397 | ] |
---|
398 | ] |
---|
399 | ] Shutting down filesystem IMFS |
---|
400 | ] *** END OF FILE SYSTEM TEST ( IMFS ) *** |
---|
401 | > Breakpoint |
---|
402 | > 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14 |
---|
403 | > 14 { |
---|
404 | Result: passed Time: 0:00:10.045447 |
---|
405 | |
---|
406 | * GDB command line (Note: paths with \'....' have been shortened) |
---|
407 | * Lines starting with ``>`` are from GDB's console. |
---|
408 | * Line starting with ``]`` are from the target's console. |
---|
409 | * The result with the test time. |
---|
410 | |
---|
411 | Running Tests in Parallel |
---|
412 | ------------------------- |
---|
413 | |
---|
414 | The RTEMS Tester supports parallel execution of tests by default. This only |
---|
415 | makes sense if the test back-end can run in parallel without resulting in |
---|
416 | resource contention. Simulators are an example of back-ends that can run in |
---|
417 | parallel. A hardware debug tool like a BDM or JTAG pod can manage only a |
---|
418 | single test at once so the tests need to be run one at a time. |
---|
419 | |
---|
420 | The test framework manages the test jobs and orders the output in the report |
---|
421 | log in test order. Output is held for completed tests until the next test to be |
---|
422 | reported has finished. |
---|
423 | |
---|
424 | Command Line Help |
---|
425 | ----------------- |
---|
426 | |
---|
427 | The :program:`rtems-test` command line accepts a range of options. You can |
---|
428 | review the available option by the ``--help`` option: |
---|
429 | |
---|
430 | .. code-block:: shell |
---|
431 | |
---|
432 | RTEMS Tools Project (c) 2012-2014 Chris Johns |
---|
433 | Options and arguments: |
---|
434 | --always-clean : Always clean the build tree, even with an error |
---|
435 | --debug-trace : Debug trace based on specific flags |
---|
436 | --dry-run : Do everything but actually run the build |
---|
437 | --force : Force the build to proceed |
---|
438 | --jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs. |
---|
439 | --keep-going : Do not stop on an error. |
---|
440 | --list-bsps : List the supported BSPs |
---|
441 | --log file : Log file where all build output is written to |
---|
442 | --macros file[,file] : Macro format files to load after the defaults |
---|
443 | --no-clean : Do not clean up the build tree |
---|
444 | --quiet : Quiet output (not used) |
---|
445 | --report-mode : Reporting modes, failures (default),all,none |
---|
446 | --rtems-bsp : The RTEMS BSP to run the test on |
---|
447 | --rtems-tools : The path to the RTEMS tools |
---|
448 | --target : Set the target triplet |
---|
449 | --timeout : Set the test timeout in seconds (default 180 seconds) |
---|
450 | --trace : Trace the execution |
---|
451 | --warn-all : Generate warnings |
---|