= RTEMS Tester Improvements = [[TOC(GSoC/2017/RTEMSTesterImprovements , depth=2)]] '''Student:''' Tanu Hari Dixit. '''Mentors:''' Kuan-Hsun Chen, Chris Johns. '''Introduction:''' RTEMS-Tester is a test framework which can execute tests not only those related to RTEMS but also for any suitable application. The RTEMS testing tool resides in the RTEMS Tools package and this project aims at enhancing and improving some of the existing functionality and to complete needed parts. '''Related Open Project Tickets:''' https://devel.rtems.org/ticket/2919 https://devel.rtems.org/ticket/2927 '''Objectives:''' The objectives of the project are: * As of now, the configuration file format is a simple text file (with .mc extension) that serves to provide all the needed macros. This is really cryptic and needs to be converted to a suitable configuration format. I believe if I could add support for configuration files in ~~YAML~~ INI format, which will make it more readable, it would come really handy for developers to understand and tune them to their needs. * Also, sometimes a BSP configuration needs some user-specific data to function as expected. For example, look at this[https://git.rtems.org/rtems-tools/tree/tester/rtems/testing/bsps/xilinx_zynq_zc706.mc#n59 1] macro file. Here, the path to the first stage bootloader (fsbl) has to be provided that is placed somewhere in the host machine. For this there should be proper control provided to the user as opposed to the hard coded configuration that is presently the case. This will help in the tunability of the configuration files. * Currently, the tests that are expected to fail, the results of which are indeterminate and those which timeout because they expect input from user, all are counted as failed (or timeouts in the last case). This masks the actual reasons why a certain test failed (or timed out) and eventuate into test statistics that can be easily misunderstood. Therefore, one objective of the project is to add support for test states “expected-fail”, “indeterminate” and “user-input”, so that they can be tracked separately. * Furthermore, regression analysis is another objective of the project. Adding YAML files for each bsp that contain an expected statistic for the tests, shall help the cause. Hence, this statistic can be compared with the current number of failed tests and help determine if the BSP has regressed against a change in rtems (or the test results in RTEMS need updating.) * The present console support in the back end of rtems-tester is given by termios[https://docs.python.org/2/library/termios.html 2]. This python library does not work on Windows[https://docs.python.org/2/library/termios.html#module-termios 3]. Hence, my goal is to integrate PySerial [https://github.com/pyserial/pyserial 4] so that the benefits of it being a cross-platform tool help support consoles on both Windows and Unix. * Ser2net is used for providing remote access to a unit’s server ports over telnet. It is proposed to add support for a telnet console in rtems-tester so that it becomes convenient to telnet into serially connected devices and test applications on them, remotely. * I’ll add simulator recipes for RTEMS to work with gem5 for sparc64/usiii and arm/realview_pbx_a9_qemu BSPs. * I also plan to export the results of rtems-tester to xml format and text-report so that it can be plotted easily and used for other analysis as developers and users deem fit. == Updates: == We have decided to rewrite configuration files in INI format instead of YAML in the absence of standard support for YAML in Python. Using PyYAML would add a dependency in the tool that would be difficult to handle across all hosts we support. === Handling macro values like %{_rtscripts}: === We will use RawConfigParser to parse the macros from INI files. It handles %(value) and %{value} both as format specification. rtemstoolkit/configuration.py has support for interpolation and extrapolation. It smoothens out the hard work of parsing ini files across hosts (and Python versions). === Handling those macro files that need user-specific things === We plan on taking assistance from a user-specified "settings" file that can help the main configuration file with user-specifics. THe ini file should have some defaults that lead to suitable errors which the user can rectify in the settings file. === Macro Maps === The macro maps are like namespaces and shouldn't be mixed for re-usability in different macro files for different BSPs. === Format === I am adding an informal explanation here. I'll soon come up with a ReST document for this :) I have segregated the macro files on the basis of their similarity (or complexity): The simple ones are (Set 1): arm7tdmi.mc \\ arm7tdmi-run.mc \\ arm920.mc \\ arm920-run.mc \\ arm1136jfs.mc \\ arm1136jfs-run.mc \\ arm1136js.mc \\ arm1136js-run.mc \\ armcortexa9.mc \\ armcortexa9-run.mc \\ erc32.mc \\ erc32-run.mc \\ h8sxsim.mc \\ h8sxsim-run.mc \\ leon2.mc \\ leon2-run.mc \\ leon3.mc \\ leon3-run.mc \\ m32csim.mc \\ m32csim-run.mc \\ simsh1.mc \\ simsh1-run.mc \\ simsh2.mc \\ simsh2-run.mc \\ simsh2e.mc \\ simsh2e-run.mc \\ simsh4.mc \\ simsh4-run.mc \\ sis.mc \\ sis-run.mc \\ v850e1sim.mc \\ v850e1sim-run.mc \\ v850e2sim.mc \\ v850e2sim-run.mc \\ v850e2v3sim.mc \\ v850e2v3sim-run.mc \\ v850esim.mc \\ v850esim-run.mc \\ v850essim.mc \\ v850essim-run.mc \\ v850sim.mc \\ v850sim-run.mc \\ We can convert the .mc file .ini on the basis of the following pattern: arm7tdmi.mc --> {{{ [global] bsp: none, none, 'arm7tdmi' [arm7tdmi] arm7tdmi: none, none, '%{_rtscripts}/gdb.cfg' arm7tdmi_arch: none, none, 'arm' gdb_script: none, none, 'arm7tdmi_gdb_script' arm7tdmi_gdb_script: none, none, '''target sim load run''' }}} arm7tdmi.ini --> {{{ [bsp] bsp = 'arm7tdmi' gdb = '%{_rtscripts}/gdb.cfg' arch = 'arm' [gdb_script] gdb_script ='arm7tdmi_gdb_script' arm7tdmi_gdb_script = 'target sim load run' }}} arm7tdmi-run.mc --> {{{ [global] bsp: none, none, 'arm7tdmi' [arm7tdmi] arm7tdmi: none, none, '%{_rtscripts}/run.cfg' arm7tdmi_arch: none, none, 'arm' bsp_run_cmd: none, none, '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run' bsp_run_opts: none, none, '-a -nouartrx' }}} arm7tdmi-run.ini --> {{{ [bsp] bsp = 'arm7tdmi' [run] run = '%{_rtscripts}/run.cfg' arch = 'arm' bsp_run_cmd = '%{rtems_tools}/%{bsp_arch}-rtems%{rtems_version}-run' bsp_run_opts = '-a -nouartrx' }}} The following macro files are similar to Set 1 except that their gdb_script is a bit different: bf537Stamp.mc \\ bf537Stamp-run.mc \\ jmr3904.mc \\ jmr3904-run.mc \\ leon3_tsim.mc \\ lm32_evr_gdbsim.mc \\ psim.mc \\ Another set is the set of macro files for simulation on qemu (Set 2): generic_or1k.mc \\ realview_pbx_a9_qemu.mc \\ xilinx_zynq_a9_qemu.mc \\ xilinx_zynq_a9_qemu_smp.mc \\ xilinx_zynq_zc706_qemu.mc \\ These can be converted as per following: generic_or1k.mc --> {{{ [global] bsp: none, none, 'generic_or1k' [generic_or1k] generic_or1k: none, none, '%{_rtscripts}/qemu.cfg' generic_or1k_arch: none, none, 'or32' generic_or1k_opts: none, none, '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M' }}} generic_or1k.ini --> {{{ [bsp] bsp = 'generic_or1k' [qemu-script] run = '%{_rtscripts}/qemu.cfg' arch = 'or32' opts = '%{qemu_opts_base} %{qemu_opts_no_net} -m 32M' }}} Next is the set of macro files which are particularly difficult, because they require user specific information (Set 3): beagleboardxm.mc \\ mcf5235.mc //might require tty settings \\ xilinx_zynq_zc706.mc \\ These can be converted as following: xilinx_zynq_zc706.mc --> {{{ [global] bsp: none, none, 'xilinx_zynq_zc706' jobs: none, none, '1' [xilinx_zynq_zc706] xilinx_zynq_zc706: none, none, '%{_rtscripts}/gdb.cfg' xilinx_zynq_zc706_arch: none, none, 'arm' #bsp_tty_dev: none, none, '/dev/cuaU0' bsp_tty_dev: none, none, '/dev/cu.SLAB_USBtoUART' gdb_script: none, none, 'xilinx_zynq_zc706_gdb_script' xilinx_zynq_zc706_gdb_script: none, none, '''target remote kaka:3333 mon load_image /home/chris/development/si/work/hydra/boot/xilinx-zynq-fsbl/build/arm-rtems4.11-xilinx_zynq_zc706/hydra-fsbl.elf 0 elf mon resume 0 mon sleep 4000 mon halt load b bsp_reset continue''' }}} xilinx_zynq_zc706.ini --> {{{ [bsp] bsp = 'xilinx_zynq_zc706' jobs = '1' [gdb-script] gdb = '%{_rtscripts}/gdb.cfg' arch = 'arm' bsp_tty_dev = '/dev/tty0' gdb_script = 'xilinx_zynq_zc706_gdb_script' xilinx_zynq_zc706_gdb_script = 'target remote kaka:3333 mon load_image /path/to/elf.elf 0 elf mon resume 0 mon sleep 4000 mon halt load b bsp_reset continue' }}} There should be no user-interaction (like keyboard input). This testing should be scripted. So the settings file can be a passed as an INI file or a paramater --with-parameter. \\ This would decrease the burden of maintaining a database for these difficult bsps and also we'll be relieved of re -asking the user. So the errors would show up and the user would know that his settings file is fishy. \\ We can highlight the line, section, value and the user needs to figure it out. If a setting is missing we raise an error. \\ We have thought of naming the section [bsp] as [erc32] or the respective bsp name. We'll include all ini files in a top level configuration file so that any change in tool (or a macro file) can be checked. \\ Important quote from Chris: "If we have 20 bsp scripts and you make a change to the way they are parsed and loaded you would need to run the tool 20 times to check all scripts. The parsing (and the tool in general) will have to be checked once this way."