source: rtems-docs/user/tools/tester.rst @ d5b631f

5
Last change on this file since d5b631f was d5b631f, checked in by Chris Johns <chrisj@…>, on 02/27/19 at 21:36:39

User: Change output block to type none.

  • Property mode set to 100644
File size: 17.0 KB
Line 
1.. SPDX-License-Identifier: CC-BY-SA-4.0
2
3.. Copyright (C) 2017 Chris Johns <chrisj@rtems.org>
4
5.. _rtems-tester-command:
6
7RTEMS Tester and Run
8====================
9
10.. index:: Tools, rtems-test, rtems-run
11
12The RTEMS Tester is a test tool that provides a command line interface to run
13test executable on supported targets. The tool provides back-end support for
14common simulators, debuggers and boot loaders. Board support package (BSP)
15configurations for RTEMS are provided and can be used to run all the tests in
16the RTEMS test suite. The tool and it's framework is not specific to RTEMS and
17can be configured to run any suitable application.
18
19RTEMS is an embedded operating system and is cross-compiled on a range of host
20machines. The executables run on the target hardware and this can vary widely
21from open source simulators, commercial simulators, debuggers with simulators,
22debuggers with hardware specific pods and devices to targe boot
23loaders. Testing RTEMS requires the cross-compiled test executable is
24transferred to the target hardware, executed and the output captured and
25returned to the test host where it is analyzed to determine the test
26result.
27
28Running the RTEMS tests on your target is very important. It provides you with
29a traceable record showing that your RTEMS version and its tools are working at
30the level the RTEMS development team expect when releasing RTEMS. Being able to
31easily run the tests and verify the results is critical in maintaining a high
32standard.
33
34Available BSP testers
35---------------------
36
37You can list the available BSP testers with:
38
39.. code-block:: none
40
41    $ rtems-test --list-bsps
42    arm920
43    beagleboardxm
44    beagleboneblack
45    jmr3904-run
46    jmr3904
47    mcf5235
48    pc
49    psim-run
50    psim
51    realview_pbx_a9_qemu
52    sis-run
53    sis
54    xilinx_zynq_a9_qemu
55    xilinx_zynq_a9_qemu_smp
56    xilinx_zynq_zc706
57    xilinx_zynq_zc706_qemu
58    xilinx_zynq_zedboard
59
60.. note:: The list is growing all the time and if your BSP is not supported we
61          encourage you to add it and submit the configuration back to the
62          project.
63
64Some of the BSPs may appear more than once in the list. These are aliased BSP
65configurations that may use a different back-end. An example is the erc32 BSP.
66There is the ``erc32`` tester which uses the GDB back-end and the
67``erc32-run`` tester which uses the ``run`` command for erc32. We will show
68how to use :program:`rtems-test` command with the erc32 BSP because it is easy
69to build an use.
70
71.. _BuildingRTEMSTests:
72
73Building RTEMS Tests
74--------------------
75
76Build the RTEMS Kernel (See :ref:`rtems-kernel`) by cloning the repository,
77running the ``bootstrap`` procedure, building and finally installing the
78kernel. Be sure to enable tests by using ``--enable-tests`` option with
79configure after running ``bootstrap``.
80
81.. code-block:: none
82
83    $ ../../rtems.git/configure --target=sparc-rtems5 \
84                        --enable-tests --enable-rtemsbsp=erc32
85    $ make
86
87Add the `-j` option to the make command with the number of cores to run a
88parallel build.
89
90Building all the tests takes time and it uses more disk so be patient. When
91finished all the tests will have been built. Some BSPs may require a post-build
92process to be run on the RTEMS ELF executable to create an image suitable for
93execution. This can be built into the configuration script and the tester will
94perform a pre-test command to covert the executable to a suitable format for
95your target.
96
97Before running all the tests it is a good idea to run the ``hello`` test. The
98``hello`` test is an RTEMS version of the classic "Hello World" example and
99running it shows you have a working tool chain and build of RTEMS ready to run
100the tests. Using the run with the ERC32 BSP the command is:
101
102.. code-block:: none
103
104    $ sparc-rtems5-run sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe
105
106    *** BEGIN OF TEST HELLO WORLD ***
107    Hello World
108    *** END OF TEST HELLO WORLD ***
109
110The run command is the GDB simulator without the GDB part.
111
112Running the example using GDB:
113
114.. code-block:: none
115
116    $ sparc-rtems5-gdb sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe
117    GNU gdb (GDB) 7.12
118    Copyright (C) 2016 Free Software Foundation, Inc.
119    License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
120    This is free software: you are free to change and redistribute it.
121    There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
122    and "show warranty" for details.
123    This GDB was configured as "--host=x86_64-linux-gnu --target=sparc-rtems5".
124    Type "show configuration" for configuration details.
125    For bug reporting instructions, please see:
126    <http://www.gnu.org/software/gdb/bugs/>.
127    Find the GDB manual and other documentation resources online at:
128    <http://www.gnu.org/software/gdb/documentation/>.
129    For help, type "help".
130    Type "apropos word" to search for commands related to "word"...
131    Reading symbols from
132    sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe...done.
133    (gdb) target sim
134    Connected to the simulator.
135    (gdb) load
136    (gdb) r
137    Starting program: sparc-rtems5/c/erc32/testsuites/samples/hello/hello.exe
138
139
140    *** BEGIN OF TEST HELLO WORLD ***
141    Hello World
142    *** END OF TEST HELLO WORLD ***
143    [Inferior 1 (process 42000) exited normally]
144    (gdb) q
145
146The command ``r`` is used to debug set break points before issuing the GDB
147``run`` command.
148
149There are currently close to 500 separate tests and you can run them all with a
150single RTEMS Tester command.
151
152Running the Tests
153-----------------
154
155The :program:`rtems-test` command line accepts a range of options. These are
156discussed later in the manual. Any command line argument without a `--` prefix
157is a test executable. You can pass more than one executable on the command
158line. If the executable is a path to a directory the directories under that
159path are searched for any file with a ``.exe`` extension. This is the default
160extension for RTEMS executables built within RTEMS.
161
162To run the erc32 tests enter the following command from the top of the erc32
163BSP build tree:
164
165.. code-block:: none
166
167    $ ~/development/rtems/test/rtems-tools.git/tester/rtems-test \
168             --log=log_erc32_run \
169             --rtems-bsp=erc32-run \
170             --rtems-tools=$HOME/development/rtems/5 \
171                 sparc-rtems5/c/erc32/testsuites/samples
172    RTEMS Testing - Tester, 5.not_released
173    [ 1/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: base_sp.exe
174    [ 2/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: capture.exe
175    [ 3/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: cdtest.exe
176    [ 4/13] p:0  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: fileio.exe
177    [ 5/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: hello.exe
178    [ 6/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: cxx_iostream.exe
179    [ 8/13] p:2  f:0  u:0  e:0  I:0  B:0  t:2  i:0  | sparc/erc32: minimum.exe
180    [ 7/13] p:2  f:0  u:0  e:0  I:0  B:0  t:2  i:0  | sparc/erc32: loopback.exe
181    [ 9/13] p:3  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: nsecs.exe
182    [10/13] p:3  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: paranoia.exe
183    [11/13] p:4  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: pppd.exe
184    [12/13] p:6  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: ticker.exe
185    [13/13] p:6  f:0  u:0  e:0  I:0  B:0  t:3  i:0  | sparc/erc32: unlimited.exe
186    Passed:         7
187    Failed:         0
188    User Input:     0
189    Expected Fail:  0
190    Indeterminate:  0
191    Benchmark:      0
192    Timeout:        5
193    Invalid:        1
194    Total:         13
195    Average test time: 0:00:27.963000
196    Testing time     : 0:06:03.519012
197
198* The RTEMS Tester's test command. In this example we are using an absolute
199  path.
200* The ``--log`` option sends the output to a log file. By default only failed
201  tests log the complete output.
202* Select the erc32 BSP and use GDB.
203* Path to the RTEMS tools so GDB can be found.
204* Path to the erc32 BSP built with all tests to run. If you add subdirectories
205  to the path specific tests can be run.
206* The output has been shortened so it fits nicely here.
207* The test results shows passes, fails, timeouts, and invalid results. In
208  this run 13 tests passed and 5 tests timed out and 1 is invalid. The
209  timeouts are probably due to the tests not having enough execute time to
210  complete. The default timeout is 180 seconds and some of the interrupt tests
211  need longer. The amount of time depends on the performance of your host CPU
212  running the simulations.
213* The output shows the average time per test and the total time taken to run
214  all the tests.
215* If the path to the testsuites was put to
216  ``sparc-rtems5/c/erc32/testsuites`` instead of
217  ``sparc-rtems5/c/erc32/testsuites/samples`` then all the executables
218  would have been tested and not just those in samples.
219
220This BSP requires the ``--rtems-tools`` option because the SPARC GDB is the
221``sparc-rtems4.11-gdb`` command that is part of the RTEMS tools. Not every BSP
222will require this option so you will need to check the specifics of the BSP
223configuration to determine if it is needed.
224
225The output you see is each test starting to run. The :program:`rtems-test`
226command by default runs multiple tests in parallel so you will see a number
227start quickly and then new tests start as others finish. The output shown here
228is from an 8 core processor so the first 8 are started in parallel and the
229status shows the order in which they actually started, which is not 1 to 8.
230
231The test start line shows the current status of the tests. The status reported
232is when the test starts and not the result of that test. A fail, timeout or
233invalid count changing means a test running before this test started failed,
234not the starting test. The status here has 7 tests passed, no failures, 5
235timeouts and 1 invalid test.
236
237.. code-block:: none
238
239    [ 5/13] p:2  f:0  u:0  e:0  I:0  B:0  t:0  i:0  | sparc/erc32: hello.exe
240
241* [ 5/13] indicates the test number, in this case test 5 of 13 tests.
242* ``p`` is the passed test count (2 in this case)
243* ``f`` is the failed test count (0 in this case)
244* ``u`` is the count for test marked as "user-input" as they expect input from
245  user
246* ``e`` is the expected-fail count (tests that are expected to fail)
247* ``I`` is the count for tests the results of which are indeterminate
248* ``B`` is the count for benchmarked tests
249* ``t`` is the timeout test count
250* ``i`` is the invalid test count.
251* ``sparc/erc32`` is the architecture and BSP names.
252* ``hello.exe`` is the executable name.
253
254The test log records all the tests and results. The reporting mode by default
255only provides the output history if a test fails, times out, or is invalid. The
256time taken by each test is also recorded.
257
258The tests must complete in a specified time or the test is marked as timed
259out. The default timeout is 3 minutes and can be globally changed using the
260``--timeout`` command line option. The time required to complete a test can
261vary. When simulators are run in parallel the time taken depends on the
262specifics of the host machine being used. A test per core is the most stable
263method even though more tests can be run than available cores. If your machine
264needs longer or you are using a VM you may need to lengthen the timeout.
265
266Test Status
267-----------
268
269Tests can be marked with one of the following:
270
271* Pass
272* Fail
273* User-input
274* Expected-fail
275* Indeterminate
276* Benchmark
277* Timeout
278* Invalid
279
280The RTEMS console or ``stdout`` output from the test is needed to determine the
281result of the test.
282
283Pass
284^^^^
285A test passes if the start and end markers are seen in the test output. The
286start marker is ``***`` and the end mark is ``*** END OF TEST``. All tests in
287the RTEMS test suite have these markers.
288
289Fail
290^^^^
291A test fails if the start marker is seen and there is no end marker.
292
293User-input
294^^^^^^^^^^
295A test marked as "user-input" as it expects input from user
296
297Expected-fail
298^^^^^^^^^^^^^
299A test that is expected to fail.
300
301Indeterminate
302^^^^^^^^^^^^^
303A test the results of which are indeterminate.
304
305Benchmark
306^^^^^^^^^
307A benchmarked test.
308
309Timeout
310^^^^^^^
311If the test does not complete within the timeout setting the test is marked as
312having timed out.
313
314Invalid
315^^^^^^^
316If no start marker is seen the test is marked as invalid. If you are testing on
317real target hardware things can sometimes go wrong and the target may not
318initialize or respond to the debugger in an expected way.
319
320Reporting
321---------
322
323The report written to the log has the following modes:
324
325* All (``all``)
326* Failures (``failures``)
327* None (``none``)
328
329The mode is controlled using the command line option ``--report-mode`` using
330the values listed above.
331
332All
333^^^
334The output of all tests is written to the log.
335
336Failures
337^^^^^^^^
338The output of the all tests that do not pass is written to the log.
339
340None
341^^^^
342No output is written to the log.
343
344The output is tagged so you can determine where it comes from. The following is
345the complete output for the In Memory File System test ``imfs_fslink.exe``
346running on a Coldfire MCF5235 using GDB and a BDM pod:
347
348.. code-block:: none
349
350    [ 11/472] p:9   f:0   t:0   i:1   | m68k/mcf5235: imfs_fslink.exe
351    > gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe
352    > Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe...
353    > done.
354    > target remote | m68k-bdm-gdbserver pipe 003-005
355    > Remote debugging using | m68k-bdm-gdbserver pipe 003-005
356    > m68k-bdm: debug module version 0
357    > m68k-bdm: detected MCF5235
358    > m68k-bdm: architecture CF5235 connected to 003-005
359    > m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282)
360    > Process 003-005 created; pid = 0
361    > 0x00006200 in ?? ()
362    > thb *0xffe254c0
363    > Hardware assisted breakpoint 1 at 0xffe254c0
364    > continue
365    > Continuing.
366    ]
367    ]
368    ] External Reset
369    ]
370    ] ColdFire MCF5235 on the BCC
371    ] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28)
372    ] Copyright 1995-2004 Freescale Semiconductor, Inc.  All Rights Reserved.
373    ]
374    ] Enter 'help' for help.
375    ]
376    > Temporary breakpoint
377    > 1, 0xffe254c0 in ?? ()
378    > load
379    > Loading section .text, size 0x147e0 lma 0x40000
380    > Loading section .data, size 0x5d0 lma 0x547e0
381    > Start address 0x40414, load size 85424
382    > Transfer rate: 10 KB/sec, 1898 bytes/write.
383    > b bsp_reset
384    > Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14.
385    > continue
386    > Continuing.
387    ] dBUG>
388    ]
389    ] *** FILE SYSTEM TEST ( IMFS ) ***
390    ] Initializing filesystem IMFS
391    ]
392    ]
393    ] *** LINK TEST ***
394    ] link creates hardlinks
395    ] test if the stat is the same
396    ] chmod and chown
397    ] unlink then stat the file
398    ] *** END OF LINK TEST ***
399    ]
400    ]
401    ] Shutting down filesystem IMFS
402    ] *** END OF FILE SYSTEM TEST ( IMFS ) ***
403    > Breakpoint
404    > 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14
405    > 14    {
406    Result: passed     Time: 0:00:10.045447
407
408* GDB command line (Note: paths with \'....' have been shortened)
409* Lines starting with ``>`` are from GDB's console.
410* Line starting with ``]`` are from the target's console.
411* The result with the test time.
412
413Running Tests in Parallel
414-------------------------
415
416The RTEMS Tester supports parallel execution of tests by default. This only
417makes sense if the test back-end can run in parallel without resulting in
418resource contention. Simulators are an example of back-ends that can run in
419parallel. A hardware debug tool like a BDM or JTAG pod can manage only a
420single test at once so the tests need to be run one at a time.
421
422The test framework manages the test jobs and orders the output in the report
423log in test order. Output is held for completed tests until the next test to be
424reported has finished.
425
426Command Line Help
427-----------------
428
429The :program:`rtems-test` command line accepts a range of options. You can
430review the available option by the ``--help`` option:
431
432.. code-block:: none
433
434    RTEMS Tools Project (c) 2012-2014 Chris Johns
435    Options and arguments:
436    --always-clean               : Always clean the build tree, even with an error
437    --debug-trace                : Debug trace based on specific flags
438    --dry-run                    : Do everything but actually run the build
439    --force                      : Force the build to proceed
440    --jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs.
441    --keep-going                 : Do not stop on an error.
442    --list-bsps                  : List the supported BSPs
443    --log file                   : Log file where all build output is written to
444    --macros file[,file]         : Macro format files to load after the defaults
445    --no-clean                   : Do not clean up the build tree
446    --quiet                      : Quiet output (not used)
447    --report-mode                : Reporting modes, failures (default),all,none
448    --rtems-bsp                  : The RTEMS BSP to run the test on
449    --rtems-tools                : The path to the RTEMS tools
450    --target                     : Set the target triplet
451    --timeout                    : Set the test timeout in seconds (default 180 seconds)
452    --trace                      : Trace the execution
453    --warn-all                   : Generate warnings
Note: See TracBrowser for help on using the repository browser.