wiki:GSoC/2012/Testing

Version 19 (modified by Chris Johns, on 12/02/14 at 10:09:49) (diff)

--

RTEMSTestingGSOC2012

Testing Framework Comparison and Sanity Check

I compared 10 testing framework by unit testing functionality and mocking capability.For each framework, I modified several tests from current test suite, compile them with RTEMS compilers, simulate the new tests with Sparc architecture, and compare the results to ensure that framework used could work correctly for RTEMS based testing.

The detailed test framework review can be accessed with this link: TestingReview

Please click on the name of each testing framework to see the detailed comparison note and sanity check result.

The coding samples can be downloaded via this link: http://code.google.com/p/rtems-testing/downloads/list

After comparison and sanity check, Unity is selected as the Framework for unity testing.

Structure Design for New Test Suite

In order to support the four testing category, the new test-suite is structured in a way that all the tests in each group belong to the same test category. The four groups of tests are Hardware Independent Tests, CPU Dependent Tests, BSP Dependent Tests, and Peripheral Dependent Tests. Tests are structured under each test group according to the functionality they serve. The areas are: Tasks(Threads) Management, Memory Management, File Systems, General(Miscellaneous) Areas, Device IO, Network, Timing Tests.

The detailed test structure design can be see via the following link: Testing Suite Plan

Conversion Plan of Current Test Suite to New Test Suite

The conversion plan contains the detailed testing plan, which is a table of the subareas within each functional area, for example, memory management. It also maps the already written tests in the current test suites to the subareas in the table. The blank element under Written Tests column indicates that there is no tests written in current test suite that exclusively tests this subarea. It does not necessarily mean that this subarea is not covered in current test suites. The tests for this subarea may be included in tests written for its parent area. Otherwise, it is not written yet.

Please access the conversion map via this link: Conversion Map

Design New Test Framework Interface

The new testing framework interface can be accessed via this link Testing Framework Interface.

This interface is designed according to Unity library. By combining some features of Google Tests, a C++ testing framework, it also allows for the future inclusion of Google Mock, which have internal preference of Google Test as Unit testing framework.

Adding JSON Output to Unity

The Unity testing library is modified to add the json output utility.

Each test suite will generate one output file naming after the test suite. For example, the test suite TestSuite.c will generate output TestSuite.c.json. For each test case, the result is an json object which has the following structure:

{ "filename":"", "functionn_name":"", "linenumber":"", "ignored":"", "result":"", "message":""

}

The code can be accessed via this link: https://github.com/panx/library.