#1549 closed defect

It must be easier to write tests

Reported by: Sebastian Huber Owned by: cynt6007
Priority: normal Milestone: 4.11
Component: build Version: 4.10
Severity: normal Keywords:
Cc: joel.sherrill@…, chrisj@…, gedare@…, cynt6007@… Blocked By:
Blocking:

Description (last modified by Chris Johns)

Unit tests are a vital development tool. It should be as easy as possible to write tests and to execute them.

Currently we have to do several steps to write a test.

  1. Create a new directory for the test.
  1. Create a new Makefile.am. This normally involves a lot of copy and paste from an existing test.
  1. Add a new init.c file. Which may be enough for simple tests.
  1. Add a test.doc file for the documentation.
  1. Add a test.scn file with the output of the test.
  1. Modify the top level Makefile.am.
  1. Modify the top level configure.ac.
  1. Run bootstrap.
  1. Add a copy and paste .cvsignore.
  1. Write a ChangeLog? with several entries.

This is way to much work. A lot of copy and paste is involved. What do we really need?

  1. An extra directory for each test case is not needed. We can use different file prefixes for each test case.
  1. Not necessary if we don't have an extra directory.
  1. This is the core.
  1. The *.doc files are pretty useless in the current format. We should define guidelines how do to document tests. We should write them in a machine and human readable form. I propose Doxygen. If someone does not like Doxygen it is trivial to write a parser that transforms the Doxygen comments into something else.
  1. These files are useless and hard to maintain. The test output should clearly state if a test failed or passed or whatever. The test output should be in a standard format to allow programs to determine the test outcome easily.
  1. We need this.
  1. We likely don't have to modify the configure.ac.
  1. We need this.
  1. We likely don't have to modify the .cvsignore.
  1. The ChangeLog? entry is now much easier.

Change History (6)

comment:1 Changed on Jun 11, 2010 at 12:35:49 AM by Chris Johns

Cc: Chris Johns added

comment:2 Changed on Jun 11, 2010 at 12:44:21 AM by Joel Sherrill

Cc: Joel Sherrill added

comment:3 Changed on Apr 6, 2013 at 1:26:42 AM by cynt6007

Owner: changed from Ralf Corsepius to cynt6007
Status: newassigned, cynt6007@vandals.uidaho.edu

comment:4 Changed on Apr 6, 2013 at 10:12:21 PM by cynt6007

Replying to comment:10:

Cynthia.. does the title of this PR " It must be easier to write tests" get
addressed in the Open Project?

Looking back over the discussion, that is the only thing that was never
discussed. We now have a set of test templates to state from and a script to
instantiate those templates as starting points. I am sure could be more
templates.

Are we trying to have a standard way to document the requirements met by a
test?

And tie the requirements to source, tests and coverage?

I copied the description into http://www.rtems.org/wiki/index.php/RTEMS_Test_Template and noted what was completed and what was left to do...

There were several comments that went into different directions, but I didn't completely capture all of those, as they were not in the description, but below is my understanding of them, and where they could be addressed...

Comment 1: looks like Sebastian Huber solved part of the problem.
Comment 2: new stuff was added, looks like requirements traceability and simulator verification of screen-shots. I should explicitly state requirements traceability in http://www.rtems.org/wiki/index.php/RTEMS_Test_Specification , and simulator verification in http://www.rtems.org/wiki/index.php/RTEMS_Test_Screen_Validation
Comment 3: a new thread related to some sort of coverage testing hook, which I thought was probably address in http://www.rtems.org/wiki/index.php/RTEMS_Test_Coverage
Comment 4: back to requirements traceability, but this time, embedded into the code. I could explicitly state requirements traceability in http://www.rtems.org/wiki/index.php/RTEMS_Test_Specification is to be embedded in the init.c
Comment 5: some vague argument related to what format the output text should be in... I'll probably write up the tests should be commented in RTEMS doxygen format and gloss over with a statement that there must exist a technology to convert doxygen comments into .texi (possibly even doxygen does this, but I'm not sure)
Comment 6: some question as to where to place comments, and the format of them, probably could be in http://www.rtems.org/wiki/index.php/RTEMS_Test_Specification

Comment 7: Is there a specific direction you wanted to go with this? There are some really good ideas for how to write up the additional issues brought up in the previous comments.

My personal take on this is the write-ups will need to be easy to understand and the results easy to maintain. Although doxygen may not be the "be all" and "end all" of documentation, it would meet Sebastian's desire to have the comments in the same file as the code. The way to do requirements traceability using doxygen would be:
Grouping by class using something akin to.
http://www.longacre-scm.com/blog/index.php/2010/08/using-doxygens-test-command-with-c
For traceability, add custom doxygen tags, such as:
@requirement (in the the source file)
and
@see (in the test file pointing to the functions being tested)
Again, if doxygen is not the format of choice, the tags should help with requirements traceability using another static analysis tool, it's just that we use doxygen in other parts of the code.

IMO the doxygen commenting (in a way towards getting tests standards compliant) of the existing tests should be a GCI task, which I will work on writing up, if that would meet the need.

comment:5 Changed on Apr 12, 2014 at 1:01:41 AM by Gedare Bloom

Cc: Gedare Bloom added
Milestone: 4.114.12

comment:6 Changed on Nov 20, 2014 at 3:26:18 AM by Chris Johns

Description: modified (diff)
Milestone: 4.124.11
Resolution: wontfix
Status: assignedclosed
Note: See TracTickets for help on using tickets.