Notice: We have migrated to GitLab launching 2024-05-01 see here:

#1549 closed defect

It must be easier to write tests

Reported by: Sebastian Huber Owned by: cynt6007
Priority: normal Milestone: 4.11
Component: build Version: 4.10
Severity: normal Keywords:
Cc: joel.sherrill@…, chrisj@…, gedare@…, cynt6007@… Blocked By:

Description (last modified by Chris Johns)

Unit tests are a vital development tool. It should be as easy as possible to write tests and to execute them.

Currently we have to do several steps to write a test.

  1. Create a new directory for the test.
  1. Create a new This normally involves a lot of copy and paste from an existing test.
  1. Add a new init.c file. Which may be enough for simple tests.
  1. Add a test.doc file for the documentation.
  1. Add a test.scn file with the output of the test.
  1. Modify the top level
  1. Modify the top level
  1. Run bootstrap.
  1. Add a copy and paste .cvsignore.
  1. Write a ChangeLog? with several entries.

This is way to much work. A lot of copy and paste is involved. What do we really need?

  1. An extra directory for each test case is not needed. We can use different file prefixes for each test case.
  1. Not necessary if we don't have an extra directory.
  1. This is the core.
  1. The *.doc files are pretty useless in the current format. We should define guidelines how do to document tests. We should write them in a machine and human readable form. I propose Doxygen. If someone does not like Doxygen it is trivial to write a parser that transforms the Doxygen comments into something else.
  1. These files are useless and hard to maintain. The test output should clearly state if a test failed or passed or whatever. The test output should be in a standard format to allow programs to determine the test outcome easily.
  1. We need this.
  1. We likely don't have to modify the
  1. We need this.
  1. We likely don't have to modify the .cvsignore.
  1. The ChangeLog? entry is now much easier.

Change History (6)

comment:1 Changed on 06/11/10 at 00:35:49 by Chris Johns

Cc: Chris Johns added

comment:2 Changed on 06/11/10 at 00:44:21 by Joel Sherrill

Cc: Joel Sherrill added

comment:3 Changed on 04/06/13 at 01:26:42 by cynt6007

Owner: changed from Ralf Corsepius to cynt6007
Status: newassigned,

comment:4 Changed on 04/06/13 at 22:12:21 by cynt6007

Replying to comment:10:

Cynthia.. does the title of this PR " It must be easier to write tests" get
addressed in the Open Project?

Looking back over the discussion, that is the only thing that was never
discussed. We now have a set of test templates to state from and a script to
instantiate those templates as starting points. I am sure could be more

Are we trying to have a standard way to document the requirements met by a

And tie the requirements to source, tests and coverage?

I copied the description into and noted what was completed and what was left to do...

There were several comments that went into different directions, but I didn't completely capture all of those, as they were not in the description, but below is my understanding of them, and where they could be addressed...

Comment 1: looks like Sebastian Huber solved part of the problem.
Comment 2: new stuff was added, looks like requirements traceability and simulator verification of screen-shots. I should explicitly state requirements traceability in , and simulator verification in
Comment 3: a new thread related to some sort of coverage testing hook, which I thought was probably address in
Comment 4: back to requirements traceability, but this time, embedded into the code. I could explicitly state requirements traceability in is to be embedded in the init.c
Comment 5: some vague argument related to what format the output text should be in... I'll probably write up the tests should be commented in RTEMS doxygen format and gloss over with a statement that there must exist a technology to convert doxygen comments into .texi (possibly even doxygen does this, but I'm not sure)
Comment 6: some question as to where to place comments, and the format of them, probably could be in

Comment 7: Is there a specific direction you wanted to go with this? There are some really good ideas for how to write up the additional issues brought up in the previous comments.

My personal take on this is the write-ups will need to be easy to understand and the results easy to maintain. Although doxygen may not be the "be all" and "end all" of documentation, it would meet Sebastian's desire to have the comments in the same file as the code. The way to do requirements traceability using doxygen would be:
Grouping by class using something akin to.
For traceability, add custom doxygen tags, such as:
@requirement (in the the source file)
@see (in the test file pointing to the functions being tested)
Again, if doxygen is not the format of choice, the tags should help with requirements traceability using another static analysis tool, it's just that we use doxygen in other parts of the code.

IMO the doxygen commenting (in a way towards getting tests standards compliant) of the existing tests should be a GCI task, which I will work on writing up, if that would meet the need.

comment:5 Changed on 04/12/14 at 01:01:41 by Gedare Bloom

Cc: Gedare Bloom added
Milestone: 4.114.12

comment:6 Changed on 11/20/14 at 03:26:18 by Chris Johns

Description: modified (diff)
Milestone: 4.124.11
Resolution: wontfix
Status: assignedclosed
Note: See TracTickets for help on using tickets.