Changeset c8f4ba6 in rtems-tools


Ignore:
Timestamp:
Jun 9, 2014, 11:41:47 PM (5 years ago)
Author:
Chris Johns <chrisj@…>
Branches:
4.10, 4.11, e9f48e9face6c4dea62712d73b35412c0fe10332
Children:
f25fed3
Parents:
984b06d
Message:

rt: Update the documentation.

File:
1 edited

Legend:

Unmodified
Added
Removed
  • doc/rtems-tester.txt

    r984b06d rc8f4ba6  
    1010
    1111Chris Johns <chrisj@rtems.org>
    12 1.0, Janurary 2014
     121.0, June 2014
    1313
    1414RTEMS Tester
     
    3333with a tracable record your RTEMS version and its tools and working at the
    3434level the RTEMS development team expect when releasing RTEMS. Being able to
    35 easly run the tests and verify the results is critical is maintiaining a high
     35easly run the tests and verify the results is critical in maintiaining a high
    3636standard.
    3737
     
    4747           section.
    4848
     49This document is for users of the RTEMS Tester as well as those wanting to
     50develop, extend and investigate the RTEMS Tester.
     51
    4952License
    5053-------
     
    7679-------------------------------------------------------------
    7780
    78 First fetch the RTEMS tester. During the development stages of this tool it is
    79 provided as a tarball. Point your browser to
    80 http://www.rtems.org/ftp/pub/rtems/people/chrisj/rtems-tester and download the
    81 latest version. The versions are dated and number if more than one release is
    82 made on a single day. Unpack the tarball and enter the directory:
    83 
    84 -------------------------------------------------------------
    85 $ tar jxf rtems-test-20140117-1.tar.bz2
    86 $ cd rtems-test
     81First fetch the RTEMS tester from the RTEMS Tools repository::
     82
     83-------------------------------------------------------------
     84$ git git://git.rtems.org/rtems-tools.git rtems-tools.git
     85$ cd rtems-tools.git/tester
    8786-------------------------------------------------------------
    8887
     
    9493-------------------------------------------------------------
    9594$ ./rtems-test --list-bsps
     95  arm920
     96  beagleboardxm
     97  jmr3904-run
     98  jmr3904
    9699  mcf5235
     100  psim-run
     101  psim
    97102  realview_pbx_a9_qemu
    98103  sis-run
    99104  sis
     105  xilinx_zynq_a9_qemu
     106  xilinx_zynq_a9_qemu_smp
    100107  xilinx_zynq_zc706
     108  xilinx_zynq_zc706_qemu
    101109-------------------------------------------------------------
    102110
    103111[TIP]
    104112=============================================================
    105 The list is growing all the time and if your BSP is not support we
    106 encourage you to add it and submit the configuration back to the project.
     113
     114The list is growing all the time and if your BSP is not supported we encourage
     115you to add it and submit the configuration back to the project.
     116
    107117=============================================================
    108118
     
    111121Instruction Simulator (SIS) BSP. There is the 'sis' BSP which uses the GDB
    112122backend and the 'sis-run' which uses the command line version of the SIS
    113 simulator. We will show how to use +rtems-test+ with the SIS BSP because it is
    114 easy to build an use.
     123simulator. We will show how to use +rtems-test+ conmand with the SIS BSP
     124because it is easy to build an use.
    115125
    116126Building RTEMS Tests
    117127~~~~~~~~~~~~~~~~~~~~
    118128
    119 Building RTEMS with a configuration command line something similar to:
     129Build RTEMS with a configuration command line something similar to:
    120130
    121131[NOTE]
     
    134144$ ./bootstrap -c && ./bootstrap -p && ./bootstrap
    135145$ cd ..
    136 $ mkdir sis
    137 $ ../rtems.git/configure --target=sparc-rtems4.11 \
    138                   --enable-tests --enable-rtemsbsp=sis
    139 $ make
    140 -------------------------------------------------------------
    141 
    142 It will build all the tests. There are currently 471 separate tests and you can
    143 run them all with a single RTEMS Tester command. Building all the tests takes
    144 time and it uses more disk so be patient.
     146$ mkdir -p builds/sis
     147$ cd builds/sis
     148$ ../../rtems.git/configure --target=sparc-rtems4.11 \
     149                    --enable-tests --enable-rtemsbsp=sis
     150$ make <1>
     151-------------------------------------------------------------
     152<1> Add the -j option with the number of cores to run the build parallel where
     153it can.
     154
     155Building all the tests takes time and it uses more disk so be patient. When
     156finished all the tests will be built ready to run. Before running all the tests
     157it is a good idea to run the +hello+ test. The +hello+ test is an RTEMS version
     158of the classic "Hello World" example and running it shows you have a working
     159tool chain and build of RTEMS ready to run the tests. Using the run command:
     160
     161-------------------------------------------------------------
     162$ sparc-rtems4.11-run <1> sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
     163
     164*** BEGIN OF TEST HELLO WORLD ***
     165Hello World
     166*** END OF TEST HELLO WORLD ***
     167-------------------------------------------------------------
     168<1> The run command is the GDB simulator without the GDB part.
     169
     170Running the example using GDB:
     171
     172-------------------------------------------------------------
     173$ sparc-rtems4.11-gdb sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
     174GNU gdb (GDB) 7.7
     175Copyright (C) 2014 Free Software Foundation, Inc.
     176License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
     177This is free software: you are free to change and redistribute it.
     178There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
     179and "show warranty" for details.
     180This GDB was configured as "--host=x86_64-apple-darwin13.2.0
     181--target=sparc-rtems4.11".
     182Type "show configuration" for configuration details.
     183For bug reporting instructions, please see:
     184<http://www.gnu.org/software/gdb/bugs/>.
     185Find the GDB manual and other documentation resources online at:
     186<http://www.gnu.org/software/gdb/documentation/>.
     187For help, type "help".
     188Type "apropos word" to search for commands related to "word"...
     189Reading symbols from
     190sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe...done.
     191(gdb) target sim
     192Connected to the simulator.
     193(gdb) load
     194(gdb) r <1>
     195Starting program: sparc-rtems4.11/c/sis/testsuites/samples/hello/hello.exe
     196
     197
     198*** BEGIN OF TEST HELLO WORLD ***
     199Hello World
     200*** END OF TEST HELLO WORLD ***
     201[Inferior 1 (process 42000) exited normally]
     202(gdb) q
     203-------------------------------------------------------------
     204<1> To debug set break points before issuing the GDB run command.
     205
     206There are currently close to 500 separate tests and you can run them all with a
     207single RTEMS Tester command.
    145208
    146209TIP: Use a recent RSB version to build a SPARC tool chain. A recent patch
     
    159222for RTEMS executables built within RTEMS.
    160223
    161 To run the SIS tests we have built run this command:
    162 
    163 -------------------------------------------------------------
    164 ./rtems-test --log=log_sis_run <1> \
    165              --rtems-bsp=sis-run <2> \
    166              --rtems-tools=$HOME/development/rtems/4.11 <3> \
    167                  $HOME/development/rtems/kernel/sis <4>
     224To run the SIS tests enter the following command from the top of the SIS BSP
     225build tree:
     226
     227-------------------------------------------------------------
     228$ ~/development/rtems/test/rtems-tools.git/tester/rtems-test <1> \
     229             --log=log_sis_run <2> \
     230             --rtems-bsp=sis-run <3> \
     231             --rtems-tools=$HOME/development/rtems/4.11 <4> \
     232                 sparc-rtems4.11/c/sis/testsuites <5>
    168233RTEMS Testing - Tester, v0.2.0
    169 [  5/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfswrite01.exe
    170 [  2/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsformat01.exe
    171 [  4/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfssync01.exe
    172 [  6/471] p:0   f:0   t:0   i:0   | sparc/sis: fsfseeko01.exe
    173 [  3/471] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsname01.exe
    174 [  1/471] p:0   f:0   t:0   i:0   | sparc/sis: fsbdpart01.exe
    175 [  7/471] p:0   f:0   t:0   i:0   | sparc/sis: fsimfsgeneric01.exe
    176 [  8/471] p:0   f:0   t:0   i:0   | sparc/sis: fsnofs01.exe
    177 [  9/471] p:1   f:0   t:0   i:0   | sparc/sis: fsrfsbitmap01.exe
    178 [ 10/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fserror.exe
    179 [ 11/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fslink.exe
    180 [ 12/471] p:4   f:0   t:0   i:0   | sparc/sis: imfs_fspatheval.exe
    181   ................................................. <5>
    182 [465/471] p:453 f:4   t:0   i:0   | sparc/sis: tm26.exe
    183 [466/471] p:454 f:4   t:0   i:0   | sparc/sis: tm27.exe
    184 [467/471] p:455 f:4   t:0   i:0   | sparc/sis: tm28.exe
    185 [468/471] p:456 f:4   t:0   i:0   | sparc/sis: tm29.exe
    186 [469/471] p:457 f:4   t:0   i:0   | sparc/sis: tm30.exe
    187 [470/471] p:458 f:4   t:0   i:0   | sparc/sis: tmck.exe
    188 [471/471] p:459 f:4   t:0   i:0   | sparc/sis: tmoverhd.exe
    189 Passed:   466 <6>
    190 Failed:     4
    191 Timeouts:   1
     234[  1/500] p:0   f:0   t:0   i:0   | sparc/sis: fsbdpart01.exe
     235[  2/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsformat01.exe
     236[  5/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfswrite01.exe
     237[  4/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfssync01.exe
     238[  7/500] p:0   f:0   t:0   i:0   | sparc/sis: fsimfsgeneric01.exe
     239[  3/500] p:0   f:0   t:0   i:0   | sparc/sis: fsdosfsname01.exe
     240[  8/500] p:0   f:0   t:0   i:0   | sparc/sis: fsnofs01.exe
     241[  6/500] p:0   f:0   t:0   i:0   | sparc/sis: fsfseeko01.exe
     242[  9/500] p:5   f:0   t:0   i:0   | sparc/sis: fsrfsbitmap01.exe
     243[ 10/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fserror.exe
     244[ 12/500] p:5   f:0   t:0   i:0   | sparc/sis: imfs_fspatheval.exe
     245  ................................................. <6>
     246[492/500] p:482 f:0   t:2   i:0   | sparc/sis: tm25.exe
     247[493/500] p:483 f:0   t:2   i:0   | sparc/sis: tm26.exe
     248[495/500] p:485 f:0   t:2   i:0   | sparc/sis: tm28.exe
     249[494/500] p:485 f:0   t:2   i:0   | sparc/sis: tm27.exe
     250[496/500] p:487 f:0   t:2   i:0   | sparc/sis: tm29.exe
     251[497/500] p:487 f:0   t:2   i:0   | sparc/sis: tm30.exe
     252[498/500] p:489 f:0   t:2   i:0   | sparc/sis: tmck.exe
     253[499/500] p:490 f:0   t:2   i:0   | sparc/sis: tmcontext01.exe
     254[500/500] p:490 f:0   t:2   i:0   | sparc/sis: tmoverhd.exe
     255Passed:   495 <7>
     256Failed:     0
     257Timeouts:   5
    192258Invalid:    0
    193 Total:    471
    194 Testing time: 0:05:29.240084 <7>
    195 -------------------------------------------------------------
    196 <1> The +--log+ option sends the output to a log file.
    197 <2> Select the SIS BSP and use the simulator run command.
    198 <3> Path to the RTEMS tools so the simulator run command can be found.
    199 <4> Path to the SIS BSP built with all tests.
    200 <5> The output has been shortened so it fits nicely here.
    201 <6> The test results. It shows passes, fails, timeouts, and invalid results.
    202 <7> The total time taken to run all the tests.
    203 
    204 This BSP requires the +--rtems-tools+ option because the SIS simulator is the
    205 +sparc-rtems4.11-run+ command that is part of the RTEMS tools. Not every BSP
     259Total:    500
     260Average test time: 0:00:00.608249 <8>
     261Testing time     : 0:05:04.124696
     262-------------------------------------------------------------
     263<1> The RTEMS Tester's test command. In this example we are using an absolute
     264path.
     265<2> The +--log+ option sends the output to a log file. By default only failed
     266tests log the complete output.
     267<3> Select the SIS BSP and use GDB.
     268<4> Path to the RTEMS tools so GDB can be found.
     269<5> Path to the SIS BSP built with all tests to run. If you add subdirectories
     270to the path specific tests can be run.
     271<6> The output has been shortened so it fits nicely here.
     272<7> The test results. It shows passes, fails, timeouts, and invalid results. In
     273this run 495 tests passed and 5 tests timedout. The timeouts are probability
     274due the tests not having enough execute time to complete. The default timeout
     275is 180 seconds and some of the interrupt tests need longer. The amount of time
     276depends on the performance of your host CPU running the simulations.
     277<8> The average time per test and the total time taken to run all the tests.
     278
     279This BSP requires the +--rtems-tools+ option because the SPARC GDB is the
     280+sparc-rtems4.11-gdb+ command that is part of the RTEMS tools. Not every BSP
    206281will require this option so you will need to check the specifics of the BSP
    207282configration to determine if it is needed.
    208283
    209284The output you see is each test starting to run. The +rtems-test+ command can
    210 run multiple SIS simulators in parallel so you will see a number start quickly
    211 and then tests start as others finish. The output shown here is from an 8 core
    212 processor so the first 8 are started in parallel and the status shows the order
    213 they actually started which is not 1 to 7.
     285run multiple SIS GDB simulations in parallel so you will see a number start
     286quickly and then tests start as others finish. The output shown here is from an
     2878 core processor so the first 8 are started in parallel and the status shows
     288the order they actually started which is not 1 to 8.
    214289
    215290The test start line shows the current status of the tests. The status reported
    216291is when the test starts and not the result of that test. A fail, timeout or
    217292invalid count changing means a test running before this test started failed,
    218 not the starting test. The status here has 453 tests pass and 4 tests fail when
    219 test tm26.exe start:
    220 
    221 -------------------------------------------------------------
    222 [465/471]<1>  p:453<2> f:4<3>  t:0<4>  i:0<5>  | sparc/sis:<6> tm26.exe<7>
    223 -------------------------------------------------------------
    224 <1> The test number, in this case test 465 of 471 tests.
     293not the starting test. The status here has 495 tests pass and no failures and 5
     294timeouts.:
     295
     296-------------------------------------------------------------
     297[295/500]<1> p:287<2> f:0<3>  t:1<4>  i:0<5> | sparc/sis:<6> sp26.exe<7>
     298-------------------------------------------------------------
     299<1> The test number, in this case test 295 of 500 tests.
    225300<2> Passed test count.
    226301<3> Failied test count.
     
    232307The test log records all the tests and results. The reporting mode by default
    233308only provides the output history if a test fails, timeouts, or is invalid. The
    234 time taken by each test is also records.
     309time taken by each test is also recorded.
    235310
    236311The tests must complete in a specified time or the test is marked as timed
     
    239314vary. When simulators are run in parallel the time taken depends on the
    240315specifics of the host machine being used. A test per core is the most stable
    241 even though more tests can be run than available cores. If your machine needs
    242 longer or you are using a VM you may need to length the time out.
     316method even though more tests can be run than available cores. If your machine
     317needs longer or you are using a VM you may need to lengthen the time out.
    243318
    244319Test Status
     
    295370The output is tagged so you can determine where it comes from. The following is
    296371the complete output for the In Memory File System test +imfs_fslink.exe+
    297 running on a Coldfire MCF5235 using a GDB and a BDM pod:
     372running on a Coldfire MCF5235 using GDB and a BDM pod:
    298373
    299374-------------------------------------------------------------
     
    369444makes sense if the test backend can run in parallel without resulting in
    370445resource contention. Simulators are an example of backends that can run in
    371 parallel.
     446parallel. A hardware debug tool like a BDM or JTAG pod can only a single test
     447at once to the tests need to be run one at a time.
    372448
    373449The test framework manages the test jobs and orders the output in the report
    374 long in test order. Output is held for completed tests until the next test to
    375 be reported has finished.
     450log in test order. Output is held for completed tests until the next test to be
     451reported has finished.
    376452
    377453Command Line Help
Note: See TracChangeset for help on using the changeset viewer.