RTEMS Testing Tool Project

[Chris Johns]

RTEMS Testing Tool Project

Contents

1. RTEMS Testing Tool Project
   1. Mentors
   2. Status
2. Introduction
3. Project
   1. Goal
   2. Prerequisite
   3. Resources
4. Tasks
   1. Tools
   2. Configuration Files
   3. Regression Analysis
   4. Generic Serial Console
   5. Telnet Console
5. Acknowledgements
6. Miscellaneous Sections
7. References

Mentors

Chris Johns

Status

Looking for funding.

Introduction

This project involves working on the RTEMS Testing Tool that resides in the RTEMS Tools package and is part of the RTEMS Tools Project. The RTEMS Testing Tool provides a consistent way to test BSPs on different hardware. The tool aims to integrate with a wide range of testing tools and methods generating a consistent report for the RTEMS test suite.

Project

The project is to enhance and improve some of the existing functionality and to complete needed parts.

Goal

• Make the configuration file format easier to use.
• Allows users to customise and change behaviour without changing configuration files.
• To be able to determine if a change in RTEMS causes a regression.
• To be able to verify simulator and real hardware results match for a BSP.
• Improve console support.

Prerequisite

• Knowledge of C and Python programming languages.
• Knowledge of host software and building packages such as simulators.
• Knowledge of debugging and debuggers.
• Knowledge of the RTEMS Tool's Toolkit.
• Requires Unix (Linux or FreeBSD) host.
• Requires modern PC hardware. Building all tests and running takes.
• Optionally a Zedboard or MicroZed? with a JTAG debugging pod.
• Optionally a WIndows 7 or Windows 10 machine.

Resources

• Current RTEMS developers.
• The existing RTEMS Testing Tool.

Tasks

The following are the tasks:

Tools

1. Build RTEMS erc32 and xilinx_zynq_a9_qemu with all tests.
2. Build QEMU with required patches to run the Zynq BSP executables.
3. Verify rtems-test runs all tests and generates a report.

Configuration Files

The rtems-test command uses the RTEMS Tool's Toolkit macro support to read macro files. This file format is a simple text version of the macro fields and is loaded into the macro table. It is not easy to read and not really suitable. It was used as a configuration file format to get something working rather than a suitable configuration file format.

The RTEMS Testing tool uses the macros internally for its data and this is not to change. It is working and has nice features.

The task is to look at replacing the raw macro format files with a YAML file that lets us define and manage the BSP back end configurations we need to support.

An important area to be considered and implemented is an interface for user specific data a BSP back end configuration needs in order to run. For example when running the tests via JTAG you need to initialise the hardware in a target specific way and this data is not part of RTEMS so a user needs to provide it to the tool to run. A command line option is one way we need implement however adding endless command line options is not reasonable so a configuration file is also needed.

The configuration file format shall allow a user to define the console interface they have. This could be a direct serial connection or a telnet console.

Regression Analysis

The RTEMS Testing tools is to provide regression analysis for BSPs that provides the expected test result data.

This task requires adding support for the expected-fail result tests can now report and to track this value.

The task also requires a YAML format file for each BSP that defines its expected test results. The rtems-test tool shall attempt to locate the file when run in regression mode inside a build tree and also when the BSP is installed and determine if the BSP has regressed or the test results in RTEMS need updating.

Generic Serial Console

The console support in the back end is termios however this is not available on Windows. This task is to integrate a BSD licensed Python serial package that provides Unix and Windows support so consoles can be supported on Windows and Unix.

Telnet Console

Add support for telnet as a console. The ser2net package is a nice way to provide a number of serial ports on a network and a telnet console provides a way for testers to use it.

Acknowledgements

None.

Miscellaneous Sections

As the project progresses, you will need to add to the RTEMS Testing section of the User manual. This section does not currently exist but it should be present even in a skeleton form.

References

• None

[Gedare Bloom]

See also #3276 for a possible idea.