wiki:Projects/GSoC/TestingReview

Version 1 (modified by Xiaochen Pan, on Jul 6, 2012 at 11:44:09 PM) (diff)

Created page with "== Testing Framework Reviews == Testing Frameworks Lists and Review CMock? CMock is a highly automated testing framework based on Unity, a unit testing framework. *Lisense..."

TestingReview

Testing Framework Reviews

Testing Frameworks Lists and Review

CMock? CMock is a highly automated testing framework based on Unity, a unit testing framework.

  • Lisense: MIT License
  • Programming Language: C
  • Maturity:
  • Projects:
  • Maintainership:
    • 3 Core Developers: Mike Karlesky, Mark VanderVoord?, Greg Williams
    • Community(according to their website): 5 Contributors, 16 Bug Finders, 4 Mentors, 2 Porters, no estimation of users provided.
    • Latest download count:

Cmockery? Cmockery is a lightweight and highly portable unit testing framework with minimum conflicts with standard C library headers. It is well suited to work for RTEMS based testing thanks its effort to avoid relying on new features of compilers, and also its lightweight and minimum-conflict headers. It also works well for Host based tests too. Unlike CMock, Cmockery has a relatively less stringent format requirements, while it requires the tester to write the test runner and mock functions by hand, while both of which can be automatically generated by scripts in Cmock. Cmockery provides convenient interface to test for memory leaks, buffer underflow and buffer overflow, all of which would cause fatal errors for single processed RTEMS system. For mock function implementations, Cmock testers only need to define the expect input and output for certain functions in unit tests, where the mock functions are generated by ruby script according to the header files, which is convenient and easy to use. However, with Cmockery, testers need to write the mock functions by hand, which is troublesome for test automation and sometimes confusing. Cmockery provides a more detailed tests report than CMock by providing the reason for tests failures.

  • License: Apache License 2.0
  • Programming Language: C
  • Maturity:
  • Projects:
  • Maintainership:
    • 4 Core Developers
    • Last release time: Sep 2008
    • Expected update: Cmockery 0.1.3
    • Community(Estimated): 93 users
    • Latest download count: 4618

Test-dept? TestDept? is unit testing framework providing stubing capabilities. TestDept? support the concept of test-suites and test-cases, which are not clearly supported by CMockery and CMock. It also have provided a setup and teardown interface to each test cases like CMock. Stub functions are similar but different concept than mocking functions. It is supported by the replace_function(functions_to_be_replace, stub_function) interface. Mocking functions can deal with the argument expectations and mocked return values for any number of times you want. While stubing functions can only provide a uni-interface. For example, you can decide that a mocking function mock_function() to return 1 at the first time of being callled and return 2 at the second time of being called. When stub technique is used, you cannot expect the same function to return different values unless you provide the inside logic basing on inputs of arguments, which is time-consuming and inconvenient for test automation. And also, because of the uni-interface of the stubbing function, it cannot specify the different expected value for certain argument each time the stubbing function is called. In conclusion, stubing functions are not as flexible and useful as mocking functions.

  • License: GNU GPL v3
  • Programming Language: C, Assembly
  • Maturity:
  • Projects:
  • Maintainership:
    • 2 Core Developer
    • Last Update Date: Nov 2010
    • Update Frequency before Nov 2010: every four month
    • Community(Estimated): 14 users
    • Latest Download count: 231

AceUnit? AceUnit? is a simple unit testing framework based on CUnit and EmbUnit?. It conforms to JUnit 4.x architecture format, where it supports the concept of test suites, test fixtures, test cases, setup and cleanup functions. It supports automatic test discovery. It has stringent test writing format in order to use the prewritten java application as Generator for test generation automation, which is very convenient. It also have the Test Runner to run the well structured tests(suites-fixtures-cases) automatically. It is well suited to work for RTEMS based and Host based testing. Although it have provided a macro for tests to run in embedded fields, it is not 100% functional.

  • License: BSD
  • Programming Language: C
  • Maturity:
  • Projects:
  • Maintainership:
    • 2 Core Developers
    • Last update date: 2009-9-30
    • Download count: 13 downloads last week

Google Test and Google Mock? Google Tests is unit testing framework conforming to xUnit architecture. Google Mock is a library that can be used to provide mocking functionality to other C++ testing frameworks as well. They can be used in combination for host based testing with powerful functionalities they provide. By defining a strict test format, Google Tests supports automatic test discovery, and distinguishes fatal and non-fatal failures, which is not provided by most test frameworks.It also provides a rich set of assertions, and even user-defined assertions for more readable tests results display. It has provided three flexible ways for user to define custom assertions, using existing boolean functions, use AssertionResult? object, and using Predicate-formatter to further customize error messages. It also provide a mechanism to verify that unit tests fails or exit with give exit code or error message in order to ensure that assertions fails as expected. In addition, it supports parameterized tests, which can be very convenient in tests automation. Test error propagation options have made it possible to write complicated unit tests without confusion. It also supports various options for running the tests,for example, tests repetition, independent tests running, tests shuffling. It supports XML test report generation. Google Mock provides a flexible mocking interface to C++ frameworks.

  • License: New BSD license
  • Programming Language: C++
  • Maturity:
  • Projects:
  • Maintainership:
    • Core Developers: 11 for Google Test, 9 for Google Mock
    • Community: 1655 users for Google Test, 673 for Google Mock.
    • Latest update data: Apr 2011
    • Download count: 86347 for Google Test, 30548 for Google Mock.

Check? Check is simple unit testing framework for test-driven development. It forks a separate process for each unit tests. Therefore, it cannot be used in RTEMS-based testing. Test cases, test fixtures, and test suites are well supported by Check with automatic test discovery to promote tests automation. It supports tests debug with a no fork mode. It has multiple output format including XML. It also have a convenient test time out feature to rule out tests take use infinite amount of time.

CU? CU is a lightweight and simple unit testing framework suited for host-based testing with enriched set of assertions. It uses separate processes for each test suites to protect the address space for each test suites. In this case, for a single process RTEMS operating system, it cannot be used. CU supports tests cases and recursive test suites for tests automation. CU supports regression testing as a plus.

  • License: GNU Lesser General Public License v3
  • Programming Language: C
  • Maturity:
  • Projects:
  • Maintainership:
    • 1 Core Developer: Daniel Fiser
    • Last update date: 2010-9-14

CxxTest? CxxTest? is a unit testing framework in C++ with xUnit architecture. For example, it supports modern tests concept like test cases, test suites, and test fixtures. Also, it supports the standard XML output. It is easy to compile and highly portable because no pre-compiled library is used. It is well suited for Host-based testing. CxxTest? support automatic test runner generation and independent test cases running. It supports mocking functions and mocking objects. As a plus point, it supports python scripts. However, it does not support C macros preprocessing, which has limit it largely in RTEMS based testing.

  • License: LGPL v2
  • Programming Language: C++
  • Maturity:
  • Projects:
  • Maintainership:
    • Current Version: CxxTest? 4.0
    • 4 Core Developers
    • Last update date: 2012-1-7
    • Download count: 114 downloads last week

CuTest? CuTest? is very simple unit testing framework. It can be used for host-based testing. Although it claims itself to be highly portable, it does not fit to work for RTEMS based testing.

  • License: GPL v2
  • Programming Language: C
  • Maturity:
  • Projects:
  • Maintainership:
    • 1 Core Developer
    • Last update date: 2010-7-22
    • Download count: 56 downloads last week

EmbUnit? EmbUnit? is unit testing framework for embedded software. It supports the testing structure with test cases, test fixture and tests suites. It supports 3 forms of test outputs, Standard Text Outputter, Compiler Outputter and XML outputter.

  • License: MIT
  • Programming language C
  • Maturity:
  • Projects:
  • Maintainership:
    • Core Developers: 2
    • Last updated date: 2004-2-10
    • Download count: 25 last week.
    • A bad review on sourceforge saying that a bug has not been fixed for a long time.