Changeset 5a801c2 in rtems-tools
- Timestamp:
- 07/22/15 11:42:13 (9 years ago)
- Branches:
- 4.10, 5, master
- Children:
- 711f715
- Parents:
- 92935ed
- git-author:
- Ric Claus <claus@…> (07/22/15 11:42:13)
- git-committer:
- Chris Johns <chrisj@…> (09/26/15 07:16:44)
- Files:
-
- 2 edited
Legend:
- Unmodified
- Added
- Removed
-
doc/rtems-tester.txt
r92935ed r5a801c2 16 16 17 17 The RTEMS Tester is a test framework. It includes a command line interface to 18 run tests on supported targets. The framework provides back end support for18 run tests on supported targets. The framework provides back-end support for 19 19 common simulators and debuggers. The board support package (BSP) configurations 20 20 for RTEMS are provided and can be used to run all the tests provided with … … 26 26 from open source simulators, commercial simulators, debuggers with simulators, 27 27 to debuggers with hardware specific pods and devices. Testing RTEMS requires 28 the cross-compiled test executable is transfer ed to the target hardware,29 executed and the output returned to the host where it is analy ised to determine28 the cross-compiled test executable is transferred to the target hardware, 29 executed and the output returned to the host where it is analyzed to determine 30 30 the test result. The RTEMS Tester provides a framework to do this. 31 31 32 32 Running all the RTEMS tests on your target is very important. It provides you 33 with a trac able record your RTEMS version and its tools andworking at the33 with a traceable record your RTEMS version and its tools are working at the 34 34 level the RTEMS development team expect when releasing RTEMS. Being able to 35 eas ly run the tests and verify the results is critical in maintiaining a high35 easily run the tests and verify the results is critical in maintaining a high 36 36 standard. 37 37 … … 40 40 * Command line tool (+rtems-test+) 41 41 * BSP Configuration scripts 42 * Back end Configuration scripts43 * Back end Python classes42 * Back-end Configuration scripts 43 * Back-end Python classes 44 44 * Python based framework 45 45 … … 55 55 The RTEMS Tester is part of the RTEMS Tools Project. The code is released under 56 56 the OSI approved The BSD 2-Clause License. It is free to use and we encourage 57 this includingoperating systems other than RTEMS.57 this, including on operating systems other than RTEMS. 58 58 59 59 The code and command line tools must retain the same names and always reference … … 65 65 The quick start will show you how to run the test suite for a BSP. It will 66 66 explain how to get the RTEMS Tester, set it up and run the tests for the SIS 67 BSP. It assumes you have a valid SPARC tool chain and built SIS BSP version of68 RTEMS. 4.11.67 BSP. It assumes you have a valid SPARC tool chain and have built the SIS BSP 68 version of RTEMS. 4.11. 69 69 70 70 Setup 71 71 ~~~~~ 72 72 73 Set up a development work space:73 Set up a development work space: 74 74 75 75 ------------------------------------------------------------- … … 82 82 83 83 ------------------------------------------------------------- 84 $ git git://git.rtems.org/rtems-tools.git rtems-tools.git84 $ git clone git://git.rtems.org/rtems-tools.git rtems-tools.git 85 85 $ cd rtems-tools.git/tester 86 86 ------------------------------------------------------------- 87 87 88 Available BSP s89 ~~~~~~~~~~~~~~ 90 91 You can list the available BSP 's with:88 Available BSP testers 89 ~~~~~~~~~~~~~~~~~~~~~ 90 91 You can list the available BSP testers with: 92 92 93 93 ------------------------------------------------------------- … … 118 118 119 119 Some of the BSPs may appear more than once in the list. These are aliased BSP 120 configuration 's that may use a different backend. An example is the SPARC121 Instruction Simulator (SIS) BSP. There is the 'sis' BSPwhich uses the GDB122 back end and the 'sis-run'which uses the command line version of the SIS123 simulator. We will show how to use +rtems-test+ co nmand with the SIS BSP120 configurations that may use a different back-end. An example is the SPARC 121 Instruction Simulator (SIS) BSP. There is the 'sis' tester which uses the GDB 122 back-end and the 'sis-run' tester which uses the command line version of the SIS 123 simulator. We will show how to use +rtems-test+ command with the SIS BSP 124 124 because it is easy to build an use. 125 125 … … 131 131 [NOTE] 132 132 ============================================================= 133 The following assumes a Unix type host andthe tools have been built with133 The following assumes a Unix-type host and that the tools have been built with 134 134 a prefix of +$HOME/development/rtems/4.11+. 135 135 ============================================================= … … 154 154 155 155 Building all the tests takes time and it uses more disk so be patient. When 156 finished all the tests will be built ready to run. Before running all the tests156 finished all the tests will be built and ready to run. Before running all the tests 157 157 it is a good idea to run the +hello+ test. The +hello+ test is an RTEMS version 158 158 of the classic "Hello World" example and running it shows you have a working … … 219 219 executable. You can pass more than one executable on the command line. If the 220 220 executable is a path to a directory the directories under that path are 221 searched for any file with a +.exe+ extension. This is the de tault extension221 searched for any file with a +.exe+ extension. This is the default extension 222 222 for RTEMS executables built within RTEMS. 223 223 … … 271 271 <6> The output has been shortened so it fits nicely here. 272 272 <7> The test results. It shows passes, fails, timeouts, and invalid results. In 273 this run 495 tests passed and 5 tests timed out. The timeouts are probability274 duethe tests not having enough execute time to complete. The default timeout273 this run 495 tests passed and 5 tests timed out. The timeouts are probably due 274 to the tests not having enough execute time to complete. The default timeout 275 275 is 180 seconds and some of the interrupt tests need longer. The amount of time 276 276 depends on the performance of your host CPU running the simulations. … … 280 280 +sparc-rtems4.11-gdb+ command that is part of the RTEMS tools. Not every BSP 281 281 will require this option so you will need to check the specifics of the BSP 282 config ration to determine if it is needed.283 284 The output you see is each test starting to run. The +rtems-test+ command can285 run multiple SIS GDB simulations in parallel so you will see a number start 286 quickly and thentests start as others finish. The output shown here is from an282 configuration to determine if it is needed. 283 284 The output you see is each test starting to run. The +rtems-test+ command by 285 default runs multiple tests in parallel so you will see a number start quickly 286 and then new tests start as others finish. The output shown here is from an 287 287 8 core processor so the first 8 are started in parallel and the status shows 288 the order they actually startedwhich is not 1 to 8.288 the order in which they actually started, which is not 1 to 8. 289 289 290 290 The test start line shows the current status of the tests. The status reported 291 291 is when the test starts and not the result of that test. A fail, timeout or 292 292 invalid count changing means a test running before this test started failed, 293 not the starting test. The status here has 495 tests pass andno failures and 5293 not the starting test. The status here has 495 tests passed, no failures and 5 294 294 timeouts.: 295 295 … … 299 299 <1> The test number, in this case test 295 of 500 tests. 300 300 <2> Passed test count. 301 <3> Fail ied test count.301 <3> Failed test count. 302 302 <4> Timeout test count. 303 303 <5> Invalid test count. … … 306 306 307 307 The test log records all the tests and results. The reporting mode by default 308 only provides the output history if a test fails, time outs, or is invalid. The308 only provides the output history if a test fails, times out, or is invalid. The 309 309 time taken by each test is also recorded. 310 310 … … 315 315 specifics of the host machine being used. A test per core is the most stable 316 316 method even though more tests can be run than available cores. If your machine 317 needs longer or you are using a VM you may need to lengthen the time 317 needs longer or you are using a VM you may need to lengthen the timeout. 318 318 319 319 Test Status … … 340 340 .Timeout 341 341 If the test does not complete within the timeout setting the test is marked as 342 timed out.342 having timed out. 343 343 344 344 .Invalid 345 345 If no start marker is seen the test is marked as invalid. If you are testing on 346 346 real target hardware things can sometimes go wrong and the target may not 347 initiali se or respond to the debugger in an expected way.347 initialize or respond to the debugger in an expected way. 348 348 349 349 Reporting … … 442 442 443 443 The RTEMS Tester supports parallel execution of tests by default. This only 444 makes sense if the test back end can run in parallel without resulting in445 resource contention. Simulators are an example of back ends that can run in446 parallel. A hardware debug tool like a BDM or JTAG pod can only a single test447 at once to the tests need to be run one at a time.444 makes sense if the test back-end can run in parallel without resulting in 445 resource contention. Simulators are an example of back-ends that can run in 446 parallel. A hardware debug tool like a BDM or JTAG pod can manage only a 447 single test at once so the tests need to be run one at a time. 448 448 449 449 The test framework manages the test jobs and orders the output in the report … … 467 467 --keep-going : Do not stop on an error. 468 468 --list-bsps : List the supported BSPs 469 --log file : Log file where all build out is written too469 --log file : Log file where all build output is written to 470 470 --macros file[,file] : Macro format files to load after the defaults 471 471 --no-clean : Do not clean up the build tree … … 480 480 ------------------------------------------------------------- 481 481 482 Develop ement482 Development 483 483 ------------ 484 484 485 485 The RTEMS Tester framework and command line tool is under active 486 development. Th isare changing, being fixed, broken and generally improved. If487 you want to help please see the Wiki page for open it mes.486 development. These are changing, being fixed, broken and generally improved. If 487 you want to help please see the Wiki page for open items. 488 488 489 489 … … 494 494 The RTEMS Tester is based on a refactored base of Python code used in the RTEMS 495 495 Source Builder. This code provided a working tested base that has been extended 496 and expanded to meet the needs ofthe RTEMS Tester. The tester uses the496 and expanded to meet the requirements for the RTEMS Tester. The tester uses the 497 497 specifics found in the various scripts and configurations in the 498 498 rtems-testing.git repo that has been accumulated over many years. The shell 499 script implementation is restricted in what can it do and per BSP scriptis a500 maintenance burden , for example the command lines and options vary between each499 script implementation is restricted in what it can do and, per BSP script, is a 500 maintenance burden. For example the command lines and options vary between each 501 501 script. -
rtemstoolkit/options.py
r92935ed r5a801c2 105 105 '--jobs=[0..n,none,half,full]': 'Run with specified number of jobs, default: num CPUs.', 106 106 '--macros file[,file]': 'Macro format files to load after the defaults', 107 '--log file': 'Log file where all build out is written too',107 '--log file': 'Log file where all build output is written to', 108 108 } 109 109 self.opts = { 'params' : [] }
Note: See TracChangeset
for help on using the changeset viewer.