Opened on 03/02/17 at 19:19:19
Closed on 10/21/18 at 04:20:12
#2918 closed project (invalid)
Conversion to new test suite.
Reported by: | Amar Takhar | Owned by: | Amar Takhar |
---|---|---|---|
Priority: | normal | Milestone: | Indefinite |
Component: | unspecified | Version: | |
Severity: | normal | Keywords: | testing, SoC |
Cc: | aadit0402@…, cpodonnell8@… | Blocked By: | |
Blocking: |
Description (last modified by Amar Takhar)
Contents
Introduction
The current test suite is a combination of different methods and techniques which makes it difficult to maintain. It also relies on screen ouput to determine valid tests.
This project involves converting the current test suite to the new format which has already been setup and prepared.
You will learn about software and hardware testing in embedded computing. The abilities learned during this project will be transferable to any area within software engineering.
Requirements
Knowledge of testing is not required. The ideal candidate should have a firm grasp of the C language and enjoys the occasional challenge. Each test conversion has varied levels of difficulty ranging from 1-10. Some tests will require refactoring of code. Light knowledge of the Python programming language is an asset.
Preparation
It would also be beneficial for anyone applying to briefly familiarise themselves with the following topics:
- Unit tests.
- Mock tests.
- Temporal tests.
- Continuous Integration
New Build System
A new build system has been written for RTEMS using the Waf build system. In order to familiarise yourself with this please see the following email thread:
https://lists.rtems.org/pipermail/devel/2015-February/009837.html
Waf
Waf is a build system written in Python. Please read over the documents to get a rough idea of how it works. It is not necessary to learn it completely just get an idea of how it is put together.
Change History (9)
comment:1 Changed on 03/02/17 at 19:32:05 by Amar Takhar
Keywords: | SoC added |
---|
comment:2 Changed on 03/20/17 at 17:21:40 by Amar Takhar
Hello to everyone interested in this project. I will be updating with further details over the next few days. If you are interested in following please add yourself to the CC list after signing up to trac.
comment:3 Changed on 03/21/17 at 13:23:23 by aditya
Cc: | aadit0402@… added |
---|
comment:4 Changed on 03/23/17 at 14:43:55 by Amar Takhar
Description: | modified (diff) |
---|
Apologies for the delay, bad timing with work I will have time now for the next few weeks to answer questions.
comment:5 Changed on 03/23/17 at 15:00:07 by Amar Takhar
Please also refer to this email posted to the users list. I will add the content here for ease:
https://lists.rtems.org/pipermail/users/2017-March/031107.html
This project involves a near ground-up rewrite of the current testsuite harness.
Most tests are trivial to convert but many will need refactoring. The new
testsuite moves away from the 'screen' based testing we do currently to an
internal, C-based approach. That is a very general and broad overview it's not
as simple as it sounds.
I do have this working already and the tests have all been renamed.
All the tests have been renamed to use a more intuitive naming scheme no more
hidden and random file names. The web portal to the test suite is up
temporarily here:
This project also involves going into the database to ensure they have been
properly renamed and fixing any that need changing.
Feel free to drill down the site. I'm working on better documentation for this
project by the 20th I was hoping for sooner but things have gotten in the way.
Anyone working on this project is not expected to convert the entire testsuite,
just as much as possible the rest of the work will involve identifying what may
be required to complete the rest which can involve refactoring of RTEMS itself.
comment:6 follow-up: 7 Changed on 03/29/17 at 06:11:00 by Sebastian Huber
The Unity testing framework is by no means good enough for the RTEMS testsuite. It lacks some important features like thread-safety at a basic level.
See also:
https://lists.rtems.org/pipermail/devel/2013-November/004965.html
Nothing changed in this respect since 2013.
I would rather use something like the Google C++ Testing Framework, however, it uses to much resources for our low level targets.
https://git.rtems.org/sebh/rtems-gtest.git/
I don't think there exists a ready to use testing framework which meets our requirements. What are our requirements by the way? This should be addressed first.
What is the specification input for the tests itself?
How is the output produced at the device level?
How it is ensured the output processing doesn't influence the test outcome in unexpected ways across targets?
How is the output produced for test suite status consumers? Maybe have a look at XO(1).
How is this synchronized with RTEMS qualification efforts?
How it is guaranteed that we don't drop test cases accidentally during the conversion? Some tests are hard to understand. Do we keep the existing test suite as is?
And so on.
It is a several man month project if you want to do this right.
comment:7 Changed on 03/29/17 at 06:23:07 by Chris Johns
Replying to Sebastian Huber:
How is the output produced for test suite status consumers? Maybe have a look at XO(1).
The only consumer on the table at the moment is rtems-test
and I would prefer we do not fragment how tests are run. The rtems-test
tool is designed to provide a consistent user interface across varying ways we test RTEMS. What happens under the hood is flexible so I hope it can be updated to handle any new test data formats.
comment:8 Changed on 03/29/17 at 16:33:38 by Cillian O'Donnell
Cc: | cpodonnell8@… added |
---|
comment:9 Changed on 10/21/18 at 04:20:12 by Amar Takhar
Resolution: | → invalid |
---|---|
Status: | assigned → closed |
Closing this due to it's irrelevance. There have been far too many chances to this I'll open a new ticket for the suite I am working on in the future.
Add SoC keyword.