source: rtems-tools/doc/rtems-tester.txt @ a28e562

4.104.11
Last change on this file since a28e562 was a28e562, checked in by Kumar Amit Mehta <gmate.amit@…>, on May 11, 2014 at 11:43:59 PM

doc: rtems-tester.txt: Add information for building tests

The test executables will not be generated unless they are built
explicitly by invoking 'make'.

  • Property mode set to 100644
File size: 16.0 KB
Line 
1RTEMS Tester
2============
3:doctype: book
4:toc2:
5:toclevels: 5
6:icons:
7:numbered:
8
9image:images/rtemswhitebg.jpg["RTEMS",width="30%"]
10
11Chris Johns <chrisj@rtems.org>
121.0, Janurary 2014
13
14RTEMS Tester
15------------
16
17The RTEMS Tester is a test framework. It includes a command line interface to
18run tests on supported targets. The framework provides backend support for
19common simulators and debuggers. The board support package (BSP) configurations
20for RTEMS are provided and can be used to run all the tests provided with
21RTEMS. The framework is not specific to RTEMS and can be configured to run any
22suitable application.
23
24RTEMS is an embedded operating system and is cross-compiled on a range of host
25machines. The executables run on the target hardware and this can vary widely
26from open source simulators, commercial simulators, debuggers with simulators,
27to debuggers with hardware specific pods and devices. Testing RTEMS requires
28the cross-compiled test executable is transfered to the target hardware,
29executed and the output returned to the host where it is analyised to determine
30the test result. The RTEMS Tester provides a framework to do this.
31
32Running all the RTEMS tests on your target is very important. It provides you
33with a tracable record your RTEMS version and its tools and working at the
34level the RTEMS development team expect when releasing RTEMS. Being able to
35easly run the tests and verify the results is critical is maintiaining a high
36standard.
37
38The RTEMS Tester contains:
39
40* Command line tool (+rtems-test+)
41* BSP Configuration scripts
42* Backend Configuration scripts
43* Backend Python classes
44* Python based framework
45
46IMPORTANT: If you have a problem please see the <<_bugs,reporting bugs>>
47           section.
48
49License
50-------
51
52The RTEMS Tester is part of the RTEMS Tools Project. The code is released under
53the OSI approved The BSD 2-Clause License. It is free to use and we encourage
54this including operating systems other than RTEMS.
55
56The code and command line tools must retain the same names and always reference
57the RTEMS Tools Project.
58
59Quick Start
60-----------
61
62The quick start will show you how to run the test suite for a BSP. It will
63explain how to get the RTEMS Tester, set it up and run the tests for the SIS
64BSP. It assumes you have a valid SPARC tool chain and built SIS BSP version of
65RTEMS. 4.11.
66
67Setup
68~~~~~
69
70Setup a development work space:
71
72-------------------------------------------------------------
73$ cd
74$ mkdir -p development/rtems/test
75$ cd development/rtems/test
76-------------------------------------------------------------
77
78First fetch the RTEMS tester. During the development stages of this tool it is
79provided as a tarball. Point your browser to
80http://www.rtems.org/ftp/pub/rtems/people/chrisj/rtems-tester and download the
81latest version. The versions are dated and number if more than one release is
82made on a single day. Unpack the tarball and enter the directory:
83
84-------------------------------------------------------------
85$ tar jxf rtems-test-20140117-1.tar.bz2
86$ cd rtems-test
87-------------------------------------------------------------
88
89Available BSPs
90~~~~~~~~~~~~~~
91
92You can list the available BSP's with:
93
94-------------------------------------------------------------
95$ ./rtems-test --list-bsps
96  mcf5235
97  realview_pbx_a9_qemu
98  sis-run
99  sis
100  xilinx_zynq_zc706
101-------------------------------------------------------------
102
103[TIP]
104=============================================================
105The list is growing all the time and if your BSP is not support we
106encourage you to add it and submit the configuration back to the project.
107=============================================================
108
109Some of the BSPs may appear more than once in the list. These are aliased BSP
110configuration's that may use a different backend. An example is the SPARC
111Instruction Simulator (SIS) BSP. There is the 'sis' BSP which uses the GDB
112backend and the 'sis-run' which uses the command line version of the SIS
113simulator. We will show how to use +rtems-test+ with the SIS BSP because it is
114easy to build an use.
115
116Building RTEMS Tests
117~~~~~~~~~~~~~~~~~~~~
118
119Building RTEMS with a configuration command line something similar to:
120
121[NOTE]
122=============================================================
123The following assumes a Unix type host and the tools have been built with
124a prefix of +$HOME/development/rtems/4.11+.
125=============================================================
126
127-------------------------------------------------------------
128$ cd
129$ export PATH=$HOME/development/rtems/4.11/bin:$PATH
130$ mkdir -p development/rtems/kernel
131$ cd development/rtems/kernel
132$ git clone git://git.rtems.org/rtems.git rtems.git
133$ cd rtems.git
134$ ./bootstrap -c && ./bootstrap -p && ./bootstrap
135$ cd ..
136$ mkdir sis
137$ ../rtems.git/configure --target=sparc-rtems4.11 \
138                  --enable-tests --enable-rtemsbsp=sis
139$ make
140-------------------------------------------------------------
141
142It will build all the tests. There are currently 471 separate tests and you can
143run them all with a single RTEMS Tester command. Building all the tests takes
144time and it uses more disk so be patient.
145
146TIP: Use a recent RSB version to build a SPARC tool chain. A recent patch
147fixes issues with the GDB simulator and it improves the test results. It is
148also recommended you use a recent RTEMS 4.11 version as the tests have been
149cleaned up to improve the results.
150
151Running the Tests
152~~~~~~~~~~~~~~~~~
153
154The +rtems-test+ command line accepts a range of options. These are discussed
155later in the manual. Any command line argument without a +--+ prefix is a test
156executable. You can pass more than one executable on the command line. If the
157executable is a path to a directory the directories under that path are
158searched for any file with a +.exe+ extension. This is the detault extension
159for RTEMS executables built within RTEMS.
160
161To run the SIS tests we have built run this command:
162
163-------------------------------------------------------------
164./rtems-test --log=log_sis_run <1> \
165             --rtems-bsp=sis-run <2> \
166             --rtems-tools=$HOME/development/rtems/4.11 <3> \
167                 $HOME/development/rtems/kernel/sis <4>
168RTEMS Testing - Tester, v0.2.0
169[  5/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfswrite01.exe
170[  2/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsformat01.exe
171[  4/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfssync01.exe
172[  6/471] p:0   f:0   t:0   i:0   | sparc/sis: fsfseeko01.exe
173[  3/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsname01.exe
174[  1/471] p:0   f:0   t:0   i:0   | sparc/sis: fsbdpart01.exe
175[  7/471] p:0   f:0   t:0   i:0   | sparc/sis: fsimfsgeneric01.exe
176[  8/471] p:0   f:0   t:0   i:0   | sparc/sis: fsnofs01.exe
177[  9/471] p:1   f:0   t:0   i:0   | sparc/sis: fsrfsbitmap01.exe
178[ 10/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fserror.exe
179[ 11/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fslink.exe
180[ 12/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fspatheval.exe
181  ................................................. <5>
182[465/471] p:453 f:4   t:0   i:0   | sparc/sis: tm26.exe
183[466/471] p:454 f:4   t:0   i:0   | sparc/sis: tm27.exe
184[467/471] p:455 f:4   t:0   i:0   | sparc/sis: tm28.exe
185[468/471] p:456 f:4   t:0   i:0   | sparc/sis: tm29.exe
186[469/471] p:457 f:4   t:0   i:0   | sparc/sis: tm30.exe
187[470/471] p:458 f:4   t:0   i:0   | sparc/sis: tmck.exe
188[471/471] p:459 f:4   t:0   i:0   | sparc/sis: tmoverhd.exe
189Passed:   466 <6>
190Failed:     4
191Timeouts:   1
192Invalid:    0
193Total:    471
194Testing time: 0:05:29.240084 <7>
195-------------------------------------------------------------
196<1> The +--log+ option sends the output to a log file.
197<2> Select the SIS BSP and use the simulator run command.
198<3> Path to the RTEMS tools so the simulator run command can be found.
199<4> Path to the SIS BSP built with all tests.
200<5> The output has been shortened so it fits nicely here.
201<6> The test results. It shows passes, fails, timeouts, and invalid results.
202<7> The total time taken to run all the tests.
203
204This BSP requires the +--rtems-tools+ option because the SIS simulator is the
205+sparc-rtems4.11-run+ command that is part of the RTEMS tools. Not every BSP
206will require this option so you will need to check the specifics of the BSP
207configration to determine if it is needed.
208
209The output you see is each test starting to run. The +rtems-test+ command can
210run multiple SIS simulators in parallel so you will see a number start quickly
211and then tests start as others finish. The output shown here is from an 8 core
212processor so the first 8 are started in parallel and the status shows the order
213they actually started which is not 1 to 7.
214
215The test start line shows the current status of the tests. The status reported
216is when the test starts and not the result of that test. A fail, timeout or
217invalid count changing means a test running before this test started failed,
218not the starting test. The status here has 453 tests pass and 4 tests fail when
219test tm26.exe start:
220
221-------------------------------------------------------------
222[465/471]<1>  p:453<2> f:4<3>  t:0<4>  i:0<5>  | sparc/sis:<6> tm26.exe<7>
223-------------------------------------------------------------
224<1> The test number, in this case test 465 of 471 tests.
225<2> Passed test count.
226<3> Failied test count.
227<4> Timeout test count.
228<5> Invalid test count.
229<6> Architecture and BSP.
230<7> Executable name.
231
232The test log records all the tests and results. The reporting mode by default
233only provides the output history if a test fails, timeouts, or is invalid. The
234time taken by each test is also records.
235
236The tests must complete in a specified time or the test is marked as timed
237out. The default timeout is 3 minutes and can be globally changed using the
238+--timeout+ command line option. The time required to complete a test can
239vary. When simulators are run in parallel the time taken depends on the
240specifics of the host machine being used. A test per core is the most stable
241even though more tests can be run than available cores. If your machine needs
242longer or you are using a VM you may need to length the time out.
243
244Test Status
245~~~~~~~~~~~
246
247Tests can be marked with one of the following:
248
249. Pass
250. Fail
251. Timeout
252. Invalid
253
254The RTEMS console or stdout output from the test is needed to determine the
255result of the test.
256
257.Pass
258A test passes if the start and end markers are seen in the test output. The
259start marker is "+\*\*\*+" and the end mark is "+\*** END OF TEST+". All tests in the
260RTEMS test suite have these markers.
261
262.Fail
263A test fails if the start marker is seen and there is no end marker.
264
265.Timeout
266If the test does not complete within the timeout setting the test is marked as
267timed out.
268
269.Invalid
270If no start marker is seen the test is marked as invalid. If you are testing on
271real target hardware things can sometimes go wrong and the target may not
272initialise or respond to the debugger in an expected way.
273
274Reporting
275~~~~~~~~~
276
277The report written to the log has the following modes:
278
279. All (+all+)
280. Failures (+failures+)
281. None (+none+)
282
283The mode is controlled using the command line option +--report-mode+ using the
284values listed above.
285
286.All
287The output of all tests is written to the log.
288
289.Failures
290The output of the all tests that do not pass is written to the log.
291
292.None
293No output is written to the log.
294
295The output is tagged so you can determine where it comes from. The following is
296the complete output for the In Memory File System test +imfs_fslink.exe+
297running on a Coldfire MCF5235 using a GDB and a BDM pod:
298
299-------------------------------------------------------------
300[ 11/472] p:9   f:0   t:0   i:1   | m68k/mcf5235: imfs_fslink.exe
301> gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe <1>
302> Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe...
303> done. <2>
304> target remote | m68k-bdm-gdbserver pipe 003-005
305> Remote debugging using | m68k-bdm-gdbserver pipe 003-005
306> m68k-bdm: debug module version 0
307> m68k-bdm: detected MCF5235
308> m68k-bdm: architecture CF5235 connected to 003-005
309> m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282)
310> Process 003-005 created; pid = 0
311> 0x00006200 in ?? ()
312> thb *0xffe254c0
313> Hardware assisted breakpoint 1 at 0xffe254c0
314> continue
315> Continuing.
316] <3>
317]
318] External Reset
319]
320] ColdFire MCF5235 on the BCC
321] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28)
322] Copyright 1995-2004 Freescale Semiconductor, Inc.  All Rights Reserved.
323]
324] Enter 'help' for help.
325]
326> Temporary breakpoint
327> 1, 0xffe254c0 in ?? ()
328> load
329> Loading section .text, size 0x147e0 lma 0x40000
330> Loading section .data, size 0x5d0 lma 0x547e0
331> Start address 0x40414, load size 85424
332> Transfer rate: 10 KB/sec, 1898 bytes/write.
333> b bsp_reset
334> Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14.
335> continue
336> Continuing.
337] dBUG>
338]
339] *** FILE SYSTEM TEST ( IMFS ) *** <4>
340] Initializing filesystem IMFS
341]
342]
343] *** LINK TEST ***
344] link creates hardlinks
345] test if the stat is the same
346] chmod and chown
347] unlink then stat the file
348] *** END OF LINK TEST ***
349]
350]
351] Shutting down filesystem IMFS
352] *** END OF FILE SYSTEM TEST ( IMFS ) *** <5>
353> Breakpoint
354> 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14
355> 14    {
356Result: passed     Time: 0:00:10.045447 <6>
357-------------------------------------------------------------
358<1> GDB command line (Note: paths with \'....' have been shortened)
359<2> Lines starting with +>+ are from GDB's console.
360<3> Line starting with +]+ are from the target's console.
361<4> The start marker.
362<5> The end marker.
363<6> The result with the test time.
364
365Running Tests in Parallel
366-------------------------
367
368The RTEMS Tester supports parallel execution of tests by default. This only
369makes sense if the test backend can run in parallel without resulting in
370resource contention. Simulators are an example of backends that can run in
371parallel.
372
373The test framework manages the test jobs and orders the output in the report
374long in test order. Output is held for completed tests until the next test to
375be reported has finished.
376
377Command Line Help
378~~~~~~~~~~~~~~~~~
379
380The +rtems-test+ command line accepts a range of options. You can review the
381available option by the +--help+ option:
382
383-------------------------------------------------------------
384RTEMS Tools Project (c) 2012-2014 Chris Johns
385Options and arguments:
386--always-clean               : Always clean the build tree, even with an error
387--debug-trace                : Debug trace based on specific flags
388--dry-run                    : Do everything but actually run the build
389--force                      : Force the build to proceed
390--jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs.
391--keep-going                 : Do not stop on an error.
392--list-bsps                  : List the supported BSPs
393--log file                   : Log file where all build out is written too
394--macros file[,file]         : Macro format files to load after the defaults
395--no-clean                   : Do not clean up the build tree
396--quiet                      : Quiet output (not used)
397--report-mode                : Reporting modes, failures (default),all,none
398--rtems-bsp                  : The RTEMS BSP to run the test on
399--rtems-tools                : The path to the RTEMS tools
400--target                     : Set the target triplet
401--timeout                    : Set the test timeout in seconds (default 180 seconds)
402--trace                      : Trace the execution
403--warn-all                   : Generate warnings
404-------------------------------------------------------------
405
406Developement
407------------
408
409The RTEMS Tester framework and command line tool is under active
410development. This are changing, being fixed, broken and generally improved. If
411you want to help please see the Wiki page for open itmes.
412
413
414
415History
416-------
417
418The RTEMS Tester is based on a refactored base of Python code used in the RTEMS
419Source Builder. This code provided a working tested base that has been extended
420and expanded to meet the needs of the RTEMS Tester. The tester uses the
421specifics found in the various scripts and configurations in the
422rtems-testing.git repo that has been accumulated over many years. The shell
423script implementation is restricted in what can it do and per BSP script is a
424maintenance burden, for example the command lines and options vary between each
425script.
Note: See TracBrowser for help on using the repository browser.