wiki:Projects/GSoC/TestingReview

Version 22 (modified by Xiaochen Pan, on Jul 8, 2012 at 9:05:37 AM) (diff)

/* Comparison Note */

TestingReview

Unit Testing Framework Reviews

CMock

CMock is a highly automated testing framework based on Unity, a unit testing framework.

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Yes | Yes | Support | No

|}

  • Lisense: MIT License
  • Programming Language: C
  • Projects:
  • Maintainership:
    • 3 Core Developers: Mike Karlesky, Mark VanderVoord?, Greg Williams
    • Community(according to their website): 5 Contributors, 16 Bug Finders, 4 Mentors, 2 Porters, no estimation of users provided.

Cmockery

Cmockery is a lightweight and highly portable unit testing framework with minimum conflicts with standard C library headers.

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Yes | No | Support | No

|}

  • License: Apache License 2.0
  • Programming Language: C
  • Projects:
    • Latest download count: 4618
  • Maintainership:
    • 4 Core Developers
    • Last release time: Sep 2008
    • Expected update: Cmockery 0.1.3
    • Community(Estimated): 93 users

AceUnit

AceUnit? is a simple unit testing framework based on CUnit and EmbUnit?.

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | Yes | Support | No

|}

  • License: BSD
  • Programming Language: C
  • Projects:
    • Download count: 13 downloads last week
  • Maintainership:
    • 2 Core Developers
    • Last update date: 2009-9-30

Test-dept

TestDept? is unit testing framework providing stubing capabilities.

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Stub | Yes | Support | No

|}

  • License: GNU GPL v3
  • Programming Language: C, Assembly
  • Projects:
    • Latest Download count: 231
  • Maintainership:
    • 2 Core Developer
    • Last Update Date: Nov 2010
    • Update Frequency before Nov 2010: every four month
    • Community(Estimated): 14 users

Google Test and Google Mock

Google Tests is unit testing framework conforming to xUnit architecture. Google Mock is a library that can be used to provide mocking functionality to other C++ testing frameworks as well. They can be used in combination for host based testing with powerful functionalities they provide.

{| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Yes | No | Not support | Yes

|}

  • License: New BSD license
  • Programming Language: C++
  • Projects:
    • Download count: 86347 for Google Test, 30548 for Google Mock.
  • Maintainership:
    • Core Developers: 11 for Google Test, 9 for Google Mock
    • Community: 1655 users for Google Test, 673 for Google Mock.
    • Latest update date: Apr 2011

Check

Check is simple unit testing framework for test-driven development. {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | Yes

|}

CU

CU is a lightweight and simple unit testing framework suited for host-based testing with enriched set of assertions. {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | Yes

|}

  • License: GNU Lesser General Public License v3
  • Programming Language: C
  • Projects:
  • Maintainership:
    • 1 Core Developer: Daniel Fiser
    • Last update date: 2010-9-14

CxxTest

CxxTest? is a unit testing framework in C++ with xUnit architecture. {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | Yes | Yes | Not Support | Yes

|}

  • License: LGPL v2
  • Programming Language: C++
  • Projects:
    • Download count: 114 downloads last week
  • Maintainership:
    • Current Version: CxxTest? 4.0
    • 4 Core Developers
    • Last update date: 2012-1-7

CuTest

CuTest? is very simple unit testing framework. {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Not Support | No

|}

  • License: GPL v2
  • Programming Language: C
  • Projects:
    • Download count: 56 downloads last week
  • Maintainership:
    • 1 Core Developer
    • Last update date: 2010-7-22

EmbUnit

EmbUnit? is unit testing framework for embedded software. {| class="wikitable"

! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output

| Yes | No | No | Support | Yes

|}

  • License: MIT
  • Programming language C
  • Projects:
    • Download count: 25 last week.
  • Maintainership:
    • Core Developers: 2
    • Last updated date: 2004-2-10
    • A bad review on sourceforge saying that a bug has not been fixed for a long time.

Comparison Note

The list is listed in selection priority order. In other words, CMock is considered to be the top competitive choice for this project. The priority is calculated basing on the following criteria: # Whether the framework supports both RTEMS-based and Host-based Testing # Whether the framework supports mocking functionality # The automation level of tests generation and running # Whether the framework supports xUnit architecture # The integration difficult of the framework and RTEMS application # The selection of output format of the framework # The stability of the test framework # The maintainership of the test framework # The popularity of the test framework, the number of downloads each week or in total

  • The frist few criteria have the higher screening weight. The followed ones are also very important. The xUnit architecture supports is equivalently important in the sense that frameworks supporting xUnit architecture have edges on better handling relationship between test cases than frameworks that doesn't support.

The RTEMS based tests are carried out with BSP=sis and CPU=sparc. For other CPUs and BSPs, further RTEMS based testing of converted current test suites would be simulated in future with the selected framework.

Google Test and CxxUnit? are frameworks of testing with programming language C++, with the large set of self-produced libs and comprehensive testing functionalities, they are considered being too big to run for RTEMS based testing, with some cpus and bsps for RTEMS having only limited memory.

Although mocking functions are compared between the frameworks, no mocking functions are constructed in the RTEMS-based testing simulation temporarily.

Even though all ten frameworks are compared in several aspects, major simulation effort is concentrated on the top five. There is no need to simulate all of them if the test results for any top five framework is satisfiable.