wiki:Projects/GSoC/TestingReview

Version 50 (modified by Xiaochen Pan, on Aug 20, 2012 at 1:29:10 AM) (diff)

/* Tests Plan */

TestingReview

Comparison Note

Selection Criteria

The list is listed in selection priority order. In other words, CMock is considered to be the top competitive choice for this project. The priority is calculated basing on the following criteria: # Whether the framework supports both RTEMS-based and Host-based Testing # Whether the framework supports mocking functionality # The automation level of tests generation and running # Whether the framework supports xUnit architecture # The integration difficult of the framework and RTEMS application # The selection of output format of the framework # The stability of the test framework # The maintainership of the test framework # The popularity of the test framework, the number of downloads each week or in total

  • The frist few criteria have the higher screening weight. The following ones are also very important. The xUnit architecture support is equivalently important in the sense that frameworks supporting xUnit architecture have edges on handling relationships between test cases over frameworks that doesn't support.

Simulation Environment

The RTEMS based tests are carried out with BSP=sis and CPU=sparc. For other CPUs and BSPs, further RTEMS based testing of converted current test suites would be simulated in future with the selected frameworks.

C++ Frameworks

Google Test and CxxUnit? are testing frameworks with programming language C++, with the large set of self-produced libs and comprehensive testing functionalities, they are considered being too big to run for RTEMS based testing, with some cpus and bsps for RTEMS having only limited memory.

Mock

Although mocking capability are compared between the frameworks, no mocking functions are constructed in the RTEMS-based test simulations temporarily.

Simulation Effort

Even though all ten frameworks are compared in the above listed aspects, major simulation effort is concentrated on the top five. There is no need to simulate all of them if the test results for any top five framework is satisfiable.

Host-based Testing

For host-based testing framework evaluation, because host based tests are tests that does not rely on rtems APIs, I build and run the tests which are written in each framework. The screenshot are showed in each framework subpage. The screenshot is displayed only to compare the readability of the test output and amount of information contained in each framework.

This is the list of converted tests for frameworks CMock, Cmockery, Aceunit and the simulation is done RTEMS based.

http://code.google.com/p/rtems-testing/downloads/list

Unit Testing Framework Reviews

CMock

CMock is a highly automated testing framework based on Unity, a unit testing framework.

CMock

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output

| Yes | Yes | Yes | Yes | Support | No

|}

  • Lisense: MIT License
  • Programming Language: C
  • Projects:
  • Maintainership:
    • 3 Core Developers: Mike Karlesky, Mark VanderVoord?, Greg Williams
    • Community(according to their website): 5 Contributors, 16 Bug Finders, 4 Mentors, 2 Porters, no estimation of users provided.

Cmockery

Cmockery is a lightweight and highly portable unit testing framework with minimum conflicts with standard C library headers.

Cmockery

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output

| Yes | Yes | No | Yes | Support | No

|}

  • License: Apache License 2.0
  • Programming Language: C
  • Projects:
    • Latest download count: 4618
  • Maintainership:
    • 4 Core Developers
    • Last release time: Sep 2008
    • Expected update: Cmockery 0.1.3
    • Community(Estimated): 93 users

AceUnit

AceUnit? is a simple unit testing framework based on CUnit and EmbUnit?.

AceUnit

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output

| Yes | No | Yes | Yes | Support | No

|}

  • License: BSD
  • Programming Language: C
  • Projects:
    • Download count: 13 downloads last week
  • Maintainership:
    • 2 Core Developers
    • Last update date: 2009-9-30

Test-dept

TestDept? is unit testing framework providing stubing capabilities.

test-dept

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output

| Yes | Stub | Yes | Yes | Support | No

|}

  • License: GNU GPL v3
  • Programming Language: C, Assembly
  • Projects:
    • Latest Download count: 231
  • Maintainership:
    • 2 Core Developer
    • Last Update Date: Nov 2010
    • Update Frequency before Nov 2010: every four month
    • Community(Estimated): 14 users

Google Test and Google Mock

Google Tests is unit testing framework conforming to xUnit architecture. Google Mock is a library that can be used to provide mocking functionality to other C++ testing frameworks as well. They can be used in combination for host based testing with powerful functionalities they provide.

Google Test Google Mock

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output

| Yes | Yes | No | Yes | Not support | Yes

|}

  • License: New BSD license
  • Programming Language: C++
  • Projects:
    • Download count: 86347 for Google Test, 30548 for Google Mock.
  • Maintainership:
    • Core Developers: 11 for Google Test, 9 for Google Mock
    • Community: 1655 users for Google Test, 673 for Google Mock.
    • Latest update date: Apr 2011

Check

Check is simple unit testing framework for test-driven development.

Check

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | Yes

|}

CU

CU is a lightweight and simple unit testing framework suited for host-based testing with enriched set of assertions.

CU {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | Yes

|}

  • License: GNU Lesser General Public License v3
  • Programming Language: C
  • Projects:
  • Maintainership:
    • 1 Core Developer: Daniel Fiser
    • Last update date: 2010-9-14

CxxTest

CxxTest? is a unit testing framework in C++ with xUnit architecture.

CxxTest

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Yes | Yes | Not Support | Yes

|}

  • License: LGPL v2
  • Programming Language: C++
  • Projects:
    • Download count: 114 downloads last week
  • Maintainership:
    • Current Version: CxxTest? 4.0
    • 4 Core Developers
    • Last update date: 2012-1-7

CuTest

CuTest? is very simple unit testing framework.

CuTest

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | No

|}

  • License: GPL v2
  • Programming Language: C
  • Projects:
    • Download count: 56 downloads last week
  • Maintainership:
    • 1 Core Developer
    • Last update date: 2010-7-22

EmbUnit

EmbUnit? is unit testing framework for embedded software.

EmbUnit

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Support | Yes

|}

  • License: MIT
  • Programming language C
  • Projects:
    • Download count: 25 last week.
  • Maintainership:
    • Core Developers: 2
    • Last updated date: 2004-2-10
    • A bad review on sourceforge saying that a bug has not been fixed for a long time.

Tests Plan

The following is a list of links to the test plan for each area. = ==Tasks Management?===

===File Systems?===

===Memory Management?===

===General Areas?===

===Device IO?===

===Network?===

===Timing Tests?===