#2918 closed project (invalid)

Conversion to new test suite.

Reported by: Amar Takhar Owned by: Amar Takhar
Priority: normal Milestone: Indefinite
Component: unspecified Version:
Severity: normal Keywords: testing, SoC
Cc: aadit0402@…, cpodonnell8@… Blocked By:
Blocking:

Description (last modified by Amar Takhar)

Contents

    1. Introduction
    2. Requirements
    3. Preparation
      1. New Build System
      2. Waf

Introduction

The current test suite is a combination of different methods and techniques which makes it difficult to maintain. It also relies on screen ouput to determine valid tests.

This project involves converting the current test suite to the new format which has already been setup and prepared.

You will learn about software and hardware testing in embedded computing. The abilities learned during this project will be transferable to any area within software engineering.

Requirements

Knowledge of testing is not required. The ideal candidate should have a firm grasp of the C language and enjoys the occasional challenge. Each test conversion has varied levels of difficulty ranging from 1-10. Some tests will require refactoring of code. Light knowledge of the Python programming language is an asset.

Preparation

It would also be beneficial for anyone applying to briefly familiarise themselves with the following topics:

  • Unit tests.
  • Mock tests.
  • Temporal tests.
  • Continuous Integration

New Build System

A new build system has been written for RTEMS using the Waf build system. In order to familiarise yourself with this please see the following email thread:

https://lists.rtems.org/pipermail/devel/2015-February/009837.html

Waf

Waf is a build system written in Python. Please read over the documents to get a rough idea of how it works. It is not necessary to learn it completely just get an idea of how it is put together.

http://waf.io/

Change History (9)

comment:1 Changed on Mar 2, 2017 at 7:32:05 PM by Amar Takhar

Keywords: SoC added

Add SoC keyword.

comment:2 Changed on Mar 20, 2017 at 5:21:40 PM by Amar Takhar

Hello to everyone interested in this project. I will be updating with further details over the next few days. If you are interested in following please add yourself to the CC list after signing up to trac.

comment:3 Changed on Mar 21, 2017 at 1:23:23 PM by aditya

Cc: aadit0402@… added

comment:4 Changed on Mar 23, 2017 at 2:43:55 PM by Amar Takhar

Description: modified (diff)

Apologies for the delay, bad timing with work I will have time now for the next few weeks to answer questions.

comment:5 Changed on Mar 23, 2017 at 3:00:07 PM by Amar Takhar

Please also refer to this email posted to the users list. I will add the content here for ease:

https://lists.rtems.org/pipermail/users/2017-March/031107.html


This project involves a near ground-up rewrite of the current testsuite harness.
Most tests are trivial to convert but many will need refactoring. The new
testsuite moves away from the 'screen' based testing we do currently to an
internal, C-based approach. That is a very general and broad overview it's not
as simple as it sounds.

I do have this working already and the tests have all been renamed.

All the tests have been renamed to use a more intuitive naming scheme no more
hidden and random file names. The web portal to the test suite is up
temporarily here:

https://test.rtems.org/

This project also involves going into the database to ensure they have been
properly renamed and fixing any that need changing.

Feel free to drill down the site. I'm working on better documentation for this
project by the 20th I was hoping for sooner but things have gotten in the way.

Anyone working on this project is not expected to convert the entire testsuite,
just as much as possible the rest of the work will involve identifying what may
be required to complete the rest which can involve refactoring of RTEMS itself.

comment:6 Changed on Mar 29, 2017 at 6:11:00 AM by Sebastian Huber

The Unity testing framework is by no means good enough for the RTEMS testsuite. It lacks some important features like thread-safety at a basic level.

See also:

https://lists.rtems.org/pipermail/devel/2013-November/004965.html

Nothing changed in this respect since 2013.

I would rather use something like the Google C++ Testing Framework, however, it uses to much resources for our low level targets.

https://git.rtems.org/sebh/rtems-gtest.git/

I don't think there exists a ready to use testing framework which meets our requirements. What are our requirements by the way? This should be addressed first.

What is the specification input for the tests itself?

How is the output produced at the device level?

How it is ensured the output processing doesn't influence the test outcome in unexpected ways across targets?

How is the output produced for test suite status consumers? Maybe have a look at XO(1).

How is this synchronized with RTEMS qualification efforts?

How it is guaranteed that we don't drop test cases accidentally during the conversion? Some tests are hard to understand. Do we keep the existing test suite as is?

And so on.

It is a several man month project if you want to do this right.

comment:7 in reply to:  6 Changed on Mar 29, 2017 at 6:23:07 AM by Chris Johns

Replying to Sebastian Huber:

How is the output produced for test suite status consumers? Maybe have a look at XO(1).

The only consumer on the table at the moment is rtems-test and I would prefer we do not fragment how tests are run. The rtems-test tool is designed to provide a consistent user interface across varying ways we test RTEMS. What happens under the hood is flexible so I hope it can be updated to handle any new test data formats.

comment:8 Changed on Mar 29, 2017 at 4:33:38 PM by Cillian O'Donnell

Cc: cpodonnell8@… added

comment:9 Changed on Oct 21, 2018 at 4:20:12 AM by Amar Takhar

Resolution: invalid
Status: assignedclosed

Closing this due to it's irrelevance. There have been far too many chances to this I'll open a new ticket for the suite I am working on in the future.

Note: See TracTickets for help on using tickets.