1 | RTEMS Tester |
---|
2 | ============ |
---|
3 | :doctype: book |
---|
4 | :toc2: |
---|
5 | :toclevels: 5 |
---|
6 | :icons: |
---|
7 | :numbered: |
---|
8 | |
---|
9 | image:images/rtemswhitebg.jpg["RTEMS",width="30%"] |
---|
10 | |
---|
11 | Chris Johns <chrisj@rtems.org> |
---|
12 | 1.0, Janurary 2014 |
---|
13 | |
---|
14 | RTEMS Tester |
---|
15 | ------------ |
---|
16 | |
---|
17 | The RTEMS Tester is a test framework. It includes a command line interface to |
---|
18 | run tests on supported targets. The framework provides backend support for |
---|
19 | common simulators and debuggers. The board support package (BSP) configurations |
---|
20 | for RTEMS are provided and can be used to run all the tests provided with |
---|
21 | RTEMS. The framework is not specific to RTEMS and can be configured to run any |
---|
22 | suitable application. |
---|
23 | |
---|
24 | RTEMS is an embedded operating system and is cross-compiled on a range of host |
---|
25 | machines. The executables run on the target hardware and this can vary widely |
---|
26 | from open source simulators, commercial simulators, debuggers with simulators, |
---|
27 | to debuggers with hardware specific pods and devices. Testing RTEMS requires |
---|
28 | the cross-compiled test executable is transfered to the target hardware, |
---|
29 | executed and the output returned to the host where it is analyised to determine |
---|
30 | the test result. The RTEMS Tester provides a framework to do this. |
---|
31 | |
---|
32 | Running all the RTEMS tests on your target is very important. It provides you |
---|
33 | with a tracable record your RTEMS version and its tools and working at the |
---|
34 | level the RTEMS development team expect when releasing RTEMS. Being able to |
---|
35 | easly run the tests and verify the results is critical is maintiaining a high |
---|
36 | standard. |
---|
37 | |
---|
38 | The RTEMS Tester contains: |
---|
39 | |
---|
40 | * Command line tool (+rtems-test+) |
---|
41 | * BSP Configuration scripts |
---|
42 | * Backend Configuration scripts |
---|
43 | * Backend Python classes |
---|
44 | * Python based framework |
---|
45 | |
---|
46 | IMPORTANT: If you have a problem please see the <<_bugs,reporting bugs>> |
---|
47 | section. |
---|
48 | |
---|
49 | License |
---|
50 | ------- |
---|
51 | |
---|
52 | The RTEMS Tester is part of the RTEMS Tools Project. The code is released under |
---|
53 | the OSI approved The BSD 2-Clause License. It is free to use and we encourage |
---|
54 | this including operating systems other than RTEMS. |
---|
55 | |
---|
56 | The code and command line tools must retain the same names and always reference |
---|
57 | the RTEMS Tools Project. |
---|
58 | |
---|
59 | Quick Start |
---|
60 | ----------- |
---|
61 | |
---|
62 | The quick start will show you how to run the test suite for a BSP. It will |
---|
63 | explain how to get the RTEMS Tester, set it up and run the tests for the SIS |
---|
64 | BSP. It assumes you have a valid SPARC tool chain and built SIS BSP version of |
---|
65 | RTEMS. 4.11. |
---|
66 | |
---|
67 | Setup |
---|
68 | ~~~~~ |
---|
69 | |
---|
70 | Setup a development work space: |
---|
71 | |
---|
72 | ------------------------------------------------------------- |
---|
73 | $ cd |
---|
74 | $ mkdir -p development/rtems/test |
---|
75 | $ cd development/rtems/test |
---|
76 | ------------------------------------------------------------- |
---|
77 | |
---|
78 | First fetch the RTEMS tester. During the development stages of this tool it is |
---|
79 | provided as a tarball. Point your browser to |
---|
80 | http://www.rtems.org/ftp/pub/rtems/people/chrisj/rtems-tester and download the |
---|
81 | latest version. The versions are dated and number if more than one release is |
---|
82 | made on a single day. Unpack the tarball and enter the directory: |
---|
83 | |
---|
84 | ------------------------------------------------------------- |
---|
85 | $ tar jxf rtems-test-20140117-1.tar.bz2 |
---|
86 | $ cd rtems-test |
---|
87 | ------------------------------------------------------------- |
---|
88 | |
---|
89 | Available BSPs |
---|
90 | ~~~~~~~~~~~~~~ |
---|
91 | |
---|
92 | You can list the available BSP's with: |
---|
93 | |
---|
94 | ------------------------------------------------------------- |
---|
95 | $ ./rtems-test --list-bsps |
---|
96 | mcf5235 |
---|
97 | realview_pbx_a9_qemu |
---|
98 | sis-run |
---|
99 | sis |
---|
100 | xilinx_zynq_zc706 |
---|
101 | ------------------------------------------------------------- |
---|
102 | |
---|
103 | [TIP] |
---|
104 | ============================================================= |
---|
105 | The list is growing all the time and if your BSP is not support we |
---|
106 | encourage you to add it and submit the configuration back to the project. |
---|
107 | ============================================================= |
---|
108 | |
---|
109 | Some of the BSPs may appear more than once in the list. These are aliased BSP |
---|
110 | configuration's that may use a different backend. An example is the SPARC |
---|
111 | Instruction Simulator (SIS) BSP. There is the 'sis' BSP which uses the GDB |
---|
112 | backend and the 'sis-run' which uses the command line version of the SIS |
---|
113 | simulator. We will show how to use +rtems-test+ with the SIS BSP because it is |
---|
114 | easy to build an use. |
---|
115 | |
---|
116 | Building RTEMS Tests |
---|
117 | ~~~~~~~~~~~~~~~~~~~~ |
---|
118 | |
---|
119 | Building RTEMS with a configuration command line something similar to: |
---|
120 | |
---|
121 | [NOTE] |
---|
122 | ============================================================= |
---|
123 | The following assumes a Unix type host and the tools have been built with |
---|
124 | a prefix of +$HOME/development/rtems/4.11+. |
---|
125 | ============================================================= |
---|
126 | |
---|
127 | ------------------------------------------------------------- |
---|
128 | $ cd |
---|
129 | $ export PATH=$HOME/development/rtems/4.11/bin:$PATH |
---|
130 | $ mkdir -p development/rtems/kernel |
---|
131 | $ cd development/rtems/kernel |
---|
132 | $ git clone git://git.rtems.org/rtems.git rtems.git |
---|
133 | $ cd rtems.git |
---|
134 | $ ./bootstrap -c && ./bootstrap -p && ./bootstrap |
---|
135 | $ cd .. |
---|
136 | $ mkdir sis |
---|
137 | $ ../rtems.git/configure --target=sparc-rtems4.11 \ |
---|
138 | --enable-tests --enable-rtemsbsp=sis |
---|
139 | $ make |
---|
140 | ------------------------------------------------------------- |
---|
141 | |
---|
142 | It will build all the tests. There are currently 471 separate tests and you can |
---|
143 | run them all with a single RTEMS Tester command. Building all the tests takes |
---|
144 | time and it uses more disk so be patient. |
---|
145 | |
---|
146 | TIP: Use a recent RSB version to build a SPARC tool chain. A recent patch |
---|
147 | fixes issues with the GDB simulator and it improves the test results. It is |
---|
148 | also recommended you use a recent RTEMS 4.11 version as the tests have been |
---|
149 | cleaned up to improve the results. |
---|
150 | |
---|
151 | Running the Tests |
---|
152 | ~~~~~~~~~~~~~~~~~ |
---|
153 | |
---|
154 | The +rtems-test+ command line accepts a range of options. These are discussed |
---|
155 | later in the manual. Any command line argument without a +--+ prefix is a test |
---|
156 | executable. You can pass more than one executable on the command line. If the |
---|
157 | executable is a path to a directory the directories under that path are |
---|
158 | searched for any file with a +.exe+ extension. This is the detault extension |
---|
159 | for RTEMS executables built within RTEMS. |
---|
160 | |
---|
161 | To run the SIS tests we have built run this command: |
---|
162 | |
---|
163 | ------------------------------------------------------------- |
---|
164 | ./rtems-test --log=log_sis_run <1> \ |
---|
165 | --rtems-bsp=sis-run <2> \ |
---|
166 | --rtems-tools=$HOME/development/rtems/4.11 <3> \ |
---|
167 | $HOME/development/rtems/kernel/sis <4> |
---|
168 | RTEMS Testing - Tester, v0.2.0 |
---|
169 | [ 5/471] p:0 f:0 t:0 i:0 | sparc/sis: fsdosfswrite01.exe |
---|
170 | [ 2/471] p:0 f:0 t:0 i:0 | sparc/sis: fsdosfsformat01.exe |
---|
171 | [ 4/471] p:0 f:0 t:0 i:0 | sparc/sis: fsdosfssync01.exe |
---|
172 | [ 6/471] p:0 f:0 t:0 i:0 | sparc/sis: fsfseeko01.exe |
---|
173 | [ 3/471] p:0 f:0 t:0 i:0 | sparc/sis: fsdosfsname01.exe |
---|
174 | [ 1/471] p:0 f:0 t:0 i:0 | sparc/sis: fsbdpart01.exe |
---|
175 | [ 7/471] p:0 f:0 t:0 i:0 | sparc/sis: fsimfsgeneric01.exe |
---|
176 | [ 8/471] p:0 f:0 t:0 i:0 | sparc/sis: fsnofs01.exe |
---|
177 | [ 9/471] p:1 f:0 t:0 i:0 | sparc/sis: fsrfsbitmap01.exe |
---|
178 | [ 10/471] p:4 f:0 t:0 i:0 | sparc/sis: imfs_fserror.exe |
---|
179 | [ 11/471] p:4 f:0 t:0 i:0 | sparc/sis: imfs_fslink.exe |
---|
180 | [ 12/471] p:4 f:0 t:0 i:0 | sparc/sis: imfs_fspatheval.exe |
---|
181 | ................................................. <5> |
---|
182 | [465/471] p:453 f:4 t:0 i:0 | sparc/sis: tm26.exe |
---|
183 | [466/471] p:454 f:4 t:0 i:0 | sparc/sis: tm27.exe |
---|
184 | [467/471] p:455 f:4 t:0 i:0 | sparc/sis: tm28.exe |
---|
185 | [468/471] p:456 f:4 t:0 i:0 | sparc/sis: tm29.exe |
---|
186 | [469/471] p:457 f:4 t:0 i:0 | sparc/sis: tm30.exe |
---|
187 | [470/471] p:458 f:4 t:0 i:0 | sparc/sis: tmck.exe |
---|
188 | [471/471] p:459 f:4 t:0 i:0 | sparc/sis: tmoverhd.exe |
---|
189 | Passed: 466 <6> |
---|
190 | Failed: 4 |
---|
191 | Timeouts: 1 |
---|
192 | Invalid: 0 |
---|
193 | Total: 471 |
---|
194 | Testing time: 0:05:29.240084 <7> |
---|
195 | ------------------------------------------------------------- |
---|
196 | <1> The +--log+ option sends the output to a log file. |
---|
197 | <2> Select the SIS BSP and use the simulator run command. |
---|
198 | <3> Path to the RTEMS tools so the simulator run command can be found. |
---|
199 | <4> Path to the SIS BSP built with all tests. |
---|
200 | <5> The output has been shortened so it fits nicely here. |
---|
201 | <6> The test results. It shows passes, fails, timeouts, and invalid results. |
---|
202 | <7> The total time taken to run all the tests. |
---|
203 | |
---|
204 | This BSP requires the +--rtems-tools+ option because the SIS simulator is the |
---|
205 | +sparc-rtems4.11-run+ command that is part of the RTEMS tools. Not every BSP |
---|
206 | will require this option so you will need to check the specifics of the BSP |
---|
207 | configration to determine if it is needed. |
---|
208 | |
---|
209 | The output you see is each test starting to run. The +rtems-test+ command can |
---|
210 | run multiple SIS simulators in parallel so you will see a number start quickly |
---|
211 | and then tests start as others finish. The output shown here is from an 8 core |
---|
212 | processor so the first 8 are started in parallel and the status shows the order |
---|
213 | they actually started which is not 1 to 7. |
---|
214 | |
---|
215 | The test start line shows the current status of the tests. The status reported |
---|
216 | is when the test starts and not the result of that test. A fail, timeout or |
---|
217 | invalid count changing means a test running before this test started failed, |
---|
218 | not the starting test. The status here has 453 tests pass and 4 tests fail when |
---|
219 | test tm26.exe start: |
---|
220 | |
---|
221 | ------------------------------------------------------------- |
---|
222 | [465/471]<1> p:453<2> f:4<3> t:0<4> i:0<5> | sparc/sis:<6> tm26.exe<7> |
---|
223 | ------------------------------------------------------------- |
---|
224 | <1> The test number, in this case test 465 of 471 tests. |
---|
225 | <2> Passed test count. |
---|
226 | <3> Failied test count. |
---|
227 | <4> Timeout test count. |
---|
228 | <5> Invalid test count. |
---|
229 | <6> Architecture and BSP. |
---|
230 | <7> Executable name. |
---|
231 | |
---|
232 | The test log records all the tests and results. The reporting mode by default |
---|
233 | only provides the output history if a test fails, timeouts, or is invalid. The |
---|
234 | time taken by each test is also records. |
---|
235 | |
---|
236 | The tests must complete in a specified time or the test is marked as timed |
---|
237 | out. The default timeout is 3 minutes and can be globally changed using the |
---|
238 | +--timeout+ command line option. The time required to complete a test can |
---|
239 | vary. When simulators are run in parallel the time taken depends on the |
---|
240 | specifics of the host machine being used. A test per core is the most stable |
---|
241 | even though more tests can be run than available cores. If your machine needs |
---|
242 | longer or you are using a VM you may need to length the time out. |
---|
243 | |
---|
244 | Test Status |
---|
245 | ~~~~~~~~~~~ |
---|
246 | |
---|
247 | Tests can be marked with one of the following: |
---|
248 | |
---|
249 | . Pass |
---|
250 | . Fail |
---|
251 | . Timeout |
---|
252 | . Invalid |
---|
253 | |
---|
254 | The RTEMS console or stdout output from the test is needed to determine the |
---|
255 | result of the test. |
---|
256 | |
---|
257 | .Pass |
---|
258 | A test passes if the start and end markers are seen in the test output. The |
---|
259 | start marker is "+\*\*\*+" and the end mark is "+\*** END OF TEST+". All tests in the |
---|
260 | RTEMS test suite have these markers. |
---|
261 | |
---|
262 | .Fail |
---|
263 | A test fails if the start marker is seen and there is no end marker. |
---|
264 | |
---|
265 | .Timeout |
---|
266 | If the test does not complete within the timeout setting the test is marked as |
---|
267 | timed out. |
---|
268 | |
---|
269 | .Invalid |
---|
270 | If no start marker is seen the test is marked as invalid. If you are testing on |
---|
271 | real target hardware things can sometimes go wrong and the target may not |
---|
272 | initialise or respond to the debugger in an expected way. |
---|
273 | |
---|
274 | Reporting |
---|
275 | ~~~~~~~~~ |
---|
276 | |
---|
277 | The report written to the log has the following modes: |
---|
278 | |
---|
279 | . All (+all+) |
---|
280 | . Failures (+failures+) |
---|
281 | . None (+none+) |
---|
282 | |
---|
283 | The mode is controlled using the command line option +--report-mode+ using the |
---|
284 | values listed above. |
---|
285 | |
---|
286 | .All |
---|
287 | The output of all tests is written to the log. |
---|
288 | |
---|
289 | .Failures |
---|
290 | The output of the all tests that do not pass is written to the log. |
---|
291 | |
---|
292 | .None |
---|
293 | No output is written to the log. |
---|
294 | |
---|
295 | The output is tagged so you can determine where it comes from. The following is |
---|
296 | the complete output for the In Memory File System test +imfs_fslink.exe+ |
---|
297 | running on a Coldfire MCF5235 using a GDB and a BDM pod: |
---|
298 | |
---|
299 | ------------------------------------------------------------- |
---|
300 | [ 11/472] p:9 f:0 t:0 i:1 | m68k/mcf5235: imfs_fslink.exe |
---|
301 | > gdb: ..../bin/m68k-rtems4.11-gdb -i=mi --nx --quiet ..../imfs_fslink.exe <1> |
---|
302 | > Reading symbols from ..../fstests/imfs_fslink/imfs_fslink.exe... |
---|
303 | > done. <2> |
---|
304 | > target remote | m68k-bdm-gdbserver pipe 003-005 |
---|
305 | > Remote debugging using | m68k-bdm-gdbserver pipe 003-005 |
---|
306 | > m68k-bdm: debug module version 0 |
---|
307 | > m68k-bdm: detected MCF5235 |
---|
308 | > m68k-bdm: architecture CF5235 connected to 003-005 |
---|
309 | > m68k-bdm: Coldfire debug module version is 0 (5206(e)/5235/5272/5282) |
---|
310 | > Process 003-005 created; pid = 0 |
---|
311 | > 0x00006200 in ?? () |
---|
312 | > thb *0xffe254c0 |
---|
313 | > Hardware assisted breakpoint 1 at 0xffe254c0 |
---|
314 | > continue |
---|
315 | > Continuing. |
---|
316 | ] <3> |
---|
317 | ] |
---|
318 | ] External Reset |
---|
319 | ] |
---|
320 | ] ColdFire MCF5235 on the BCC |
---|
321 | ] Firmware v3b.1a.1a (Built on Jul 21 2004 17:31:28) |
---|
322 | ] Copyright 1995-2004 Freescale Semiconductor, Inc. All Rights Reserved. |
---|
323 | ] |
---|
324 | ] Enter 'help' for help. |
---|
325 | ] |
---|
326 | > Temporary breakpoint |
---|
327 | > 1, 0xffe254c0 in ?? () |
---|
328 | > load |
---|
329 | > Loading section .text, size 0x147e0 lma 0x40000 |
---|
330 | > Loading section .data, size 0x5d0 lma 0x547e0 |
---|
331 | > Start address 0x40414, load size 85424 |
---|
332 | > Transfer rate: 10 KB/sec, 1898 bytes/write. |
---|
333 | > b bsp_reset |
---|
334 | > Breakpoint 2 at 0x41274: file ..../shared/bspreset_loop.c, line 14. |
---|
335 | > continue |
---|
336 | > Continuing. |
---|
337 | ] dBUG> |
---|
338 | ] |
---|
339 | ] *** FILE SYSTEM TEST ( IMFS ) *** <4> |
---|
340 | ] Initializing filesystem IMFS |
---|
341 | ] |
---|
342 | ] |
---|
343 | ] *** LINK TEST *** |
---|
344 | ] link creates hardlinks |
---|
345 | ] test if the stat is the same |
---|
346 | ] chmod and chown |
---|
347 | ] unlink then stat the file |
---|
348 | ] *** END OF LINK TEST *** |
---|
349 | ] |
---|
350 | ] |
---|
351 | ] Shutting down filesystem IMFS |
---|
352 | ] *** END OF FILE SYSTEM TEST ( IMFS ) *** <5> |
---|
353 | > Breakpoint |
---|
354 | > 2, bsp_reset () at ..../m68k/mcf5235/../../shared/bspreset_loop.c:14 |
---|
355 | > 14 { |
---|
356 | Result: passed Time: 0:00:10.045447 <6> |
---|
357 | ------------------------------------------------------------- |
---|
358 | <1> GDB command line (Note: paths with \'....' have been shortened) |
---|
359 | <2> Lines starting with +>+ are from GDB's console. |
---|
360 | <3> Line starting with +]+ are from the target's console. |
---|
361 | <4> The start marker. |
---|
362 | <5> The end marker. |
---|
363 | <6> The result with the test time. |
---|
364 | |
---|
365 | Running Tests in Parallel |
---|
366 | ------------------------- |
---|
367 | |
---|
368 | The RTEMS Tester supports parallel execution of tests by default. This only |
---|
369 | makes sense if the test backend can run in parallel without resulting in |
---|
370 | resource contention. Simulators are an example of backends that can run in |
---|
371 | parallel. |
---|
372 | |
---|
373 | The test framework manages the test jobs and orders the output in the report |
---|
374 | long in test order. Output is held for completed tests until the next test to |
---|
375 | be reported has finished. |
---|
376 | |
---|
377 | Command Line Help |
---|
378 | ~~~~~~~~~~~~~~~~~ |
---|
379 | |
---|
380 | The +rtems-test+ command line accepts a range of options. You can review the |
---|
381 | available option by the +--help+ option: |
---|
382 | |
---|
383 | ------------------------------------------------------------- |
---|
384 | RTEMS Tools Project (c) 2012-2014 Chris Johns |
---|
385 | Options and arguments: |
---|
386 | --always-clean : Always clean the build tree, even with an error |
---|
387 | --debug-trace : Debug trace based on specific flags |
---|
388 | --dry-run : Do everything but actually run the build |
---|
389 | --force : Force the build to proceed |
---|
390 | --jobs=[0..n,none,half,full] : Run with specified number of jobs, default: num CPUs. |
---|
391 | --keep-going : Do not stop on an error. |
---|
392 | --list-bsps : List the supported BSPs |
---|
393 | --log file : Log file where all build out is written too |
---|
394 | --macros file[,file] : Macro format files to load after the defaults |
---|
395 | --no-clean : Do not clean up the build tree |
---|
396 | --quiet : Quiet output (not used) |
---|
397 | --report-mode : Reporting modes, failures (default),all,none |
---|
398 | --rtems-bsp : The RTEMS BSP to run the test on |
---|
399 | --rtems-tools : The path to the RTEMS tools |
---|
400 | --target : Set the target triplet |
---|
401 | --timeout : Set the test timeout in seconds (default 180 seconds) |
---|
402 | --trace : Trace the execution |
---|
403 | --warn-all : Generate warnings |
---|
404 | ------------------------------------------------------------- |
---|
405 | |
---|
406 | Developement |
---|
407 | ------------ |
---|
408 | |
---|
409 | The RTEMS Tester framework and command line tool is under active |
---|
410 | development. This are changing, being fixed, broken and generally improved. If |
---|
411 | you want to help please see the Wiki page for open itmes. |
---|
412 | |
---|
413 | |
---|
414 | |
---|
415 | History |
---|
416 | ------- |
---|
417 | |
---|
418 | The RTEMS Tester is based on a refactored base of Python code used in the RTEMS |
---|
419 | Source Builder. This code provided a working tested base that has been extended |
---|
420 | and expanded to meet the needs of the RTEMS Tester. The tester uses the |
---|
421 | specifics found in the various scripts and configurations in the |
---|
422 | rtems-testing.git repo that has been accumulated over many years. The shell |
---|
423 | script implementation is restricted in what can it do and per BSP script is a |
---|
424 | maintenance burden, for example the command lines and options vary between each |
---|
425 | script. |
---|