source: rtems-tools/doc/rtems-tester.txt @ 15a3e06

5
Last change on this file since 15a3e06 was 711f715, checked in by Ric Claus <claus@…>, on 07/22/15 at 12:03:06

rtems-test.txt: Some more clarity improvements.

  • Property mode set to 100644
File size: 19.1 KB
Line 
1RTEMS Tester
2============
3:doctype: book
4:toc2:
5:toclevels: 5
6:icons:
7:numbered:
8
9image:images/rtemswhitebg.jpg["RTEMS",width="30%"]
10
11Chris Johns <chrisj@rtems.org>
121.0, June 2014
13
14RTEMS Tester
15------------
16
17The RTEMS Tester is a test framework. It includes a command line interface to
18run tests on supported targets. The framework provides back-end support for
19common simulators and debuggers. The board support package (BSP) configurations
20for RTEMS are provided and can be used to run all the tests provided with
21RTEMS. The framework is not specific to RTEMS and can be configured to run any
22suitable application.
23
24RTEMS is an embedded operating system and is cross-compiled on a range of host
25machines. The executables run on the target hardware and this can vary widely
26from open source simulators, commercial simulators, debuggers with simulators,
27to debuggers with hardware specific pods and devices. Testing RTEMS requires
28the cross-compiled test executable is transferred to the target hardware,
29executed and the output returned to the host where it is analyzed to determine
30the test result. The RTEMS Tester provides a framework to do this.
31
32Running all the RTEMS tests on your target is very important. It provides you
33with a traceable record showing that your RTEMS version and its tools are
34working at the level the RTEMS development team expect when releasing
35RTEMS. Being able to easily run the tests and verify the results is critical in
36maintaining a high standard.
37
38The RTEMS Tester contains:
39
40* Command line tool (+rtems-test+)
41* BSP Configuration scripts
42* Back-end Configuration scripts
43* Back-end Python classes
44* Python based framework
45
46IMPORTANT: If you have a problem please see the <<_bugs,reporting bugs>>
47           section.
48
49This document is for users of the RTEMS Tester as well as those wanting to
50develop, extend and investigate the RTEMS Tester.
51
52License
53-------
54
55The RTEMS Tester is part of the RTEMS Tools Project. The code is released under
56the OSI approved "The BSD 2-Clause License". It is free to use and we encourage
57this, including on operating systems other than RTEMS.
58
59The code and command line tools must retain the same names and always reference
60the RTEMS Tools Project.
61
62Quick Start
63-----------
64
65The quick start will show you how to run the test suite for a BSP. It will
66explain how to get the RTEMS Tester, set it up and run the tests for the SIS
67BSP. It assumes you have a valid SPARC tool chain and have built the SIS BSP
68version of RTEMS 4.11.
69
70Setup
71~~~~~
72
73Set up a development work space:
74
75-------------------------------------------------------------
76$ cd
77$ mkdir -p development/rtems/test
78$ cd development/rtems/test
79-------------------------------------------------------------
80
81First fetch the RTEMS tester from the RTEMS Tools repository:
82
83-------------------------------------------------------------
84$ git clone git://git.rtems.org/rtems-tools.git rtems-tools.git
85$ cd rtems-tools.git/tester
86-------------------------------------------------------------
87
88Available BSP testers
89~~~~~~~~~~~~~~~~~~~~~
90
91You can list the available BSP testers with:
92
93-------------------------------------------------------------
94$ ./rtems-test --list-bsps
95  arm920
96  beagleboardxm
97  jmr3904-run
98  jmr3904
99  mcf5235
100  psim-run
101  psim
102  realview_pbx_a9_qemu
103  sis-run
104  sis
105  xilinx_zynq_a9_qemu
106  xilinx_zynq_a9_qemu_smp
107  xilinx_zynq_zc706
108  xilinx_zynq_zc706_qemu
109-------------------------------------------------------------
110
111[TIP]
112=============================================================
113
114The list is growing all the time and if your BSP is not supported we encourage
115you to add it and submit the configuration back to the project.
116
117=============================================================
118
119Some of the BSPs may appear more than once in the list. These are aliased BSP
120configurations that may use a different back-end. An example is the SPARC
121Instruction Simulator (SIS) BSP. There is the 'sis' tester which uses the GDB
122back-end and the 'sis-run' tester which uses the command line version of the SIS
123simulator. We will show how to use +rtems-test+ command with the SIS BSP
124because it is easy to build an use.
125
126Building RTEMS Tests
127~~~~~~~~~~~~~~~~~~~~
128
129Build RTEMS with a configuration command line something similar to:
130
131[NOTE]
132=============================================================
133The following assumes a Unix-type host and that the tools have been built with
134a prefix of +$HOME/development/rtems/4.11+.
135=============================================================
136
137-------------------------------------------------------------
138$ cd
139$ export PATH=$HOME/development/rtems/4.11/bin:$PATH
140$ mkdir -p development/rtems/kernel
141$ cd development/rtems/kernel
142$ git clone git://git.rtems.org/rtems.git rtems.git
143$ cd rtems.git
144$ ./bootstrap -c && ./bootstrap -p && ./bootstrap
145$ cd ..
146$ mkdir -p builds/sis
147$ cd builds/sis
148$ ../../rtems.git/configure --target=sparc-rtems4.11 \
149                    --enable-tests --enable-rtemsbsp=sis
150$ make <1>
151-------------------------------------------------------------
152<1> Add the -j option with the number of cores to run the build parallel where
153it can.
154
155Building all the tests takes time and it uses more disk so be patient. When
156finished all the tests will be built and ready to run. Before running all the tests
157it is a good idea to run the +hello+ test. The +hello+ test is an RTEMS version
158of the classic "Hello World" example and running it shows you have a working
159tool chain and build of RTEMS ready to run the tests. Using the run command:
160
161-------------------------------------------------------------
162$ sparc-rtems4.11-run <1> sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
163
164*** BEGIN OF TEST HELLO WORLD ***
165Hello World
166*** END OF TEST HELLO WORLD ***
167-------------------------------------------------------------
168<1> The run command is the GDB simulator without the GDB part.
169
170Running the example using GDB:
171
172-------------------------------------------------------------
173$ sparc-rtems4.11-gdb sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
174GNU gdb (GDB) 7.7
175Copyright (C) 2014 Free Software Foundation, Inc.
176License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
177This is free software: you are free to change and redistribute it.
178There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
179and "show warranty" for details.
180This GDB was configured as "--host=x86_64-apple-darwin13.2.0
181--target=sparc-rtems4.11".
182Type "show configuration" for configuration details.
183For bug reporting instructions, please see:
184<http://www.gnu.org/software/gdb/bugs/>.
185Find the GDB manual and other documentation resources online at:
186<http://www.gnu.org/software/gdb/documentation/>.
187For help, type "help".
188Type "apropos word" to search for commands related to "word"...
189Reading symbols from
190sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe...done.
191(gdb) target sim
192Connected to the simulator.
193(gdb) load
194(gdb) r <1>
195Starting program: sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
196
197
198*** BEGIN OF TEST HELLO WORLD ***
199Hello World
200*** END OF TEST HELLO WORLD ***
201[Inferior 1 (process 42000) exited normally]
202(gdb) q
203-------------------------------------------------------------
204<1> To debug set break points before issuing the GDB run command.
205
206There are currently close to 500 separate tests and you can run them all with a
207single RTEMS Tester command.
208
209TIP: Use a recent RSB version to build a SPARC tool chain. A recent patch
210fixes issues with the GDB simulator and it improves the test results. It is
211also recommended you use a recent RTEMS 4.11 version as the tests have been
212cleaned up to improve the results.
213
214Running the Tests
215~~~~~~~~~~~~~~~~~
216
217The +rtems-test+ command line accepts a range of options. These are discussed
218later in the manual. Any command line argument without a +--+ prefix is a test
219executable. You can pass more than one executable on the command line. If the
220executable is a path to a directory the directories under that path are
221searched for any file with a +.exe+ extension. This is the default extension
222for RTEMS executables built within RTEMS.
223
224To run the SIS tests enter the following command from the top of the SIS BSP
225build tree:
226
227-------------------------------------------------------------
228$ ~/development/rtems/test/rtems-tools.git/tester/rtems-test <1> \
229             --log=log_sis_run <2> \
230             --rtems-bsp=sis-run <3> \
231             --rtems-tools=$HOME/development/rtems/4.11 <4> \
232                 sparc-rtems4.11/c/sis/testsuites <5>
233RTEMS Testing - Tester, v0.2.0
234[  1/500] p:0   f:0   t:0   i:0   | sparc/sis: fsbdpart01.exe
235[  2/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsformat01.exe
236[  5/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfswrite01.exe
237[  4/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfssync01.exe
238[  7/500] p:0   f:0   t:0   i:0   | sparc/sis: fsimfsgeneric01.exe
239[  3/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsname01.exe
240[  8/500] p:0   f:0   t:0   i:0   | sparc/sis: fsnofs01.exe
241[  6/500] p:0   f:0   t:0   i:0   | sparc/sis: fsfseeko01.exe
242[  9/500] p:5   f:0   t:0   i:0   | sparc/sis: fsrfsbitmap01.exe
243[ 10/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fserror.exe
244[ 12/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fspatheval.exe
245  ................................................. <6>
246[492/500] p:482 f:0   t:2   i:0   | sparc/sis: tm25.exe
247[493/500] p:483 f:0   t:2   i:0   | sparc/sis: tm26.exe
248[495/500] p:485 f:0   t:2   i:0   | sparc/sis: tm28.exe
249[494/500] p:485 f:0   t:2   i:0   | sparc/sis: tm27.exe
250[496/500] p:487 f:0   t:2   i:0   | sparc/sis: tm29.exe
251[497/500] p:487 f:0   t:2   i:0   | sparc/sis: tm30.exe
252[498/500] p:489 f:0   t:2   i:0   | sparc/sis: tmck.exe
253[499/500] p:490 f:0   t:2   i:0   | sparc/sis: tmcontext01.exe
254[500/500] p:490 f:0   t:2   i:0   | sparc/sis: tmoverhd.exe
255Passed:   495 <7>
256Failed:     0
257Timeouts:   5
258Invalid:    0
259Total:    500
260Average test time: 0:00:00.608249 <8>
261Testing time     : 0:05:04.124696
262-------------------------------------------------------------
263<1> The RTEMS Tester's test command. In this example we are using an absolute
264path.
265<2> The +--log+ option sends the output to a log file. By default only failed
266tests log the complete output.
267<3> Select the SIS BSP and use GDB.
268<4> Path to the RTEMS tools so GDB can be found.
269<5> Path to the SIS BSP built with all tests to run. If you add subdirectories
270to the path specific tests can be run.
271<6> The output has been shortened so it fits nicely here.
272<7> The test results. It shows passes, fails, timeouts, and invalid results. In
273this run 495 tests passed and 5 tests timed out. The timeouts are probably due
274to the tests not having enough execute time to complete. The default timeout
275is 180 seconds and some of the interrupt tests need longer. The amount of time
276depends on the performance of your host CPU running the simulations.
277<8> The average time per test and the total time taken to run all the tests.
278
279This BSP requires the +--rtems-tools+ option because the SPARC GDB is the
280+sparc-rtems4.11-gdb+ command that is part of the RTEMS tools. Not every BSP
281will require this option so you will need to check the specifics of the BSP
282configuration to determine if it is needed.
283
284The output you see is each test starting to run. The +rtems-test+ command by
285default runs multiple tests in parallel so you will see a number start quickly
286and then new tests start as others finish. The output shown here is from an
2878 core processor so the first 8 are started in parallel and the status shows
288the order in which they actually started, which is not 1 to 8.
289
290The test start line shows the current status of the tests. The status reported
291is when the test starts and not the result of that test. A fail, timeout or
292invalid count changing means a test running before this test started failed,
293not the starting test. The status here has 495 tests passed, no failures and 5
294timeouts.:
295
296-------------------------------------------------------------
297[295/500]<1> p:287<2> f:0<3>  t:1<4>  i:0<5> | sparc/sis:<6> sp26.exe<7>
298-------------------------------------------------------------
299<1> The test number, in this case test 295 of 500 tests.
300<2> Passed test count.
301<3> Failed test count.
302<4> Timeout test count.
303<5> Invalid test count.
304<6> Architecture and BSP.
305<7> Executable name.
306
307The test log records all the tests and results. The reporting mode by default
308only provides the output history if a test fails, times out, or is invalid. The
309time taken by each test is also recorded.
310
311The tests must complete in a specified time or the test is marked as timed
312out. The default timeout is 3 minutes and can be globally changed using the
313+--timeout+ command line option. The time required to complete a test can
314vary. When simulators are run in parallel the time taken depends on the
315specifics of the host machine being used. A test per core is the most stable
316method even though more tests can be run than available cores. If your machine
317needs longer or you are using a VM you may need to lengthen the timeout.
318
319Test Status
320~~~~~~~~~~~
321
322Tests can be marked with one of the following:
323
324. Pass
325. Fail
326. Timeout
327. Invalid
328
329The RTEMS console or stdout output from the test is needed to determine the
330result of the test.
331
332.Pass
333A test passes if the start and end markers are seen in the test output. The
334start marker is "+\*\*\*+" and the end mark is "+\*** END OF TEST+". All tests in the
335RTEMS test suite have these markers.
336
337.Fail
338A test fails if the start marker is seen and there is no end marker.
339
340.Timeout
341If the test does not complete within the timeout setting the test is marked as
342having timed out.
343
344.Invalid
345If no start marker is seen the test is marked as invalid. If you are testing on
346real target hardware things can sometimes go wrong and the target may not
347initialize or respond to the debugger in an expected way.
348
349Reporting
350~~~~~~~~~
351
352The report written to the log has the following modes:
353
354. All (+all+)
355. Failures (+failures+)
356. None (+none+)
357
358The mode is controlled using the command line option +--report-mode+ using the
359values listed above.
360
361.All
362The output of all tests is written to the log.
363
364.Failures
365The output of the all tests that do not pass is written to the log.
366
367.None
368No output is written to the log.
369
370The output is tagged so you can determine where it comes from. The following is
371the complete output for the In Memory File System test +imfs_fslink.exe+
372running on a Coldfire MCF5235 using GDB and a BDM pod:
373
374-------------------------------------------------------------
375[ 11/472] p:9   f:0   t:0   i:1   | m68k/mcf5235: imfs_fslink.exe
376> gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe <1>
377> Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe...
378> done. <2>
379> target remote | m68k-bdm-gdbserver pipe 003-005
380> Remote debugging using | m68k-bdm-gdbserver pipe 003-005
381> m68k-bdm: debug module version 0
382> m68k-bdm: detected MCF5235
383> m68k-bdm: architecture CF5235 connected to 003-005
384> m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282)
385> Process 003-005 created; pid = 0
386> 0x00006200 in ?? ()
387> thb *0xffe254c0
388> Hardware assisted breakpoint 1 at 0xffe254c0
389> continue
390> Continuing.
391] <3>
392]
393] External Reset
394]
395] ColdFire MCF5235 on the BCC
396] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28)
397] Copyright 1995-2004 Freescale Semiconductor, Inc.  All Rights Reserved.
398]
399] Enter 'help' for help.
400]
401> Temporary breakpoint
402> 1, 0xffe254c0 in ?? ()
403> load
404> Loading section .text, size 0x147e0 lma 0x40000
405> Loading section .data, size 0x5d0 lma 0x547e0
406> Start address 0x40414, load size 85424
407> Transfer rate: 10 KB/sec, 1898 bytes/write.
408> b bsp_reset
409> Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14.
410> continue
411> Continuing.
412] dBUG>
413]
414] *** FILE SYSTEM TEST ( IMFS ) *** <4>
415] Initializing filesystem IMFS
416]
417]
418] *** LINK TEST ***
419] link creates hardlinks
420] test if the stat is the same
421] chmod and chown
422] unlink then stat the file
423] *** END OF LINK TEST ***
424]
425]
426] Shutting down filesystem IMFS
427] *** END OF FILE SYSTEM TEST ( IMFS ) *** <5>
428> Breakpoint
429> 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14
430> 14    {
431Result: passed     Time: 0:00:10.045447 <6>
432-------------------------------------------------------------
433<1> GDB command line (Note: paths with \'....' have been shortened)
434<2> Lines starting with +>+ are from GDB's console.
435<3> Line starting with +]+ are from the target's console.
436<4> The start marker.
437<5> The end marker.
438<6> The result with the test time.
439
440Running Tests in Parallel
441-------------------------
442
443The RTEMS Tester supports parallel execution of tests by default. This only
444makes sense if the test back-end can run in parallel without resulting in
445resource contention. Simulators are an example of back-ends that can run in
446parallel. A hardware debug tool like a BDM or JTAG pod can manage only a
447single test at once so the tests need to be run one at a time.
448
449The test framework manages the test jobs and orders the output in the report
450log in test order. Output is held for completed tests until the next test to be
451reported has finished.
452
453Command Line Help
454~~~~~~~~~~~~~~~~~
455
456The +rtems-test+ command line accepts a range of options. You can review the
457available option by the +--help+ option:
458
459-------------------------------------------------------------
460RTEMS Tools Project (c) 2012-2014 Chris Johns
461Options and arguments:
462--always-clean               : Always clean the build tree, even with an error
463--debug-trace                : Debug trace based on specific flags
464--dry-run                    : Do everything but actually run the build
465--force                      : Force the build to proceed
466--jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs.
467--keep-going                 : Do not stop on an error.
468--list-bsps                  : List the supported BSPs
469--log file                   : Log file where all build output is written to
470--macros file[,file]         : Macro format files to load after the defaults
471--no-clean                   : Do not clean up the build tree
472--quiet                      : Quiet output (not used)
473--report-mode                : Reporting modes, failures (default),all,none
474--rtems-bsp                  : The RTEMS BSP to run the test on
475--rtems-tools                : The path to the RTEMS tools
476--target                     : Set the target triplet
477--timeout                    : Set the test timeout in seconds (default 180 seconds)
478--trace                      : Trace the execution
479--warn-all                   : Generate warnings
480-------------------------------------------------------------
481
482Development
483------------
484
485The RTEMS Tester framework and command line tool is under active
486development. These are changing, being fixed, broken and generally improved. If
487you want to help please see the Wiki page for open items.
488
489
490
491History
492-------
493
494The RTEMS Tester is based on a refactored base of Python code used in the RTEMS
495Source Builder. This code provided a working tested base that has been extended
496and expanded to meet the requirements for the RTEMS Tester. The tester uses the
497specifics found in the various scripts and configurations in the
498rtems-testing.git repo that has been accumulated over many years. The shell
499script implementation is restricted in what it can do and, per BSP script, is a
500maintenance burden. For example the command lines and options vary between each
501script.
Note: See TracBrowser for help on using the repository browser.