= TestingReview = = Unit Testing Framework Reviews = = [wiki:Projects/GSoC/TestingReview/CMock CMock] = CMock is a highly automated testing framework based on Unity, a unit testing framework. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output |- | Yes | Yes | Yes | Yes | Support | No |- |} * Lisense: MIT License * Programming Language: C * Projects: * Maintainership: * 3 Core Developers: Mike Karlesky, Mark VanderVoord, Greg Williams * Community(according to their website): 5 Contributors, 16 Bug Finders, 4 Mentors, 2 Porters, no estimation of users provided. = [wiki:Projects/GSoC/TestingReview/Cmockery Cmockery] = Cmockery is a lightweight and highly portable unit testing framework with minimum conflicts with standard C library headers. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output |- | Yes | Yes | No | Yes | Support | No |- |} * License: Apache License 2.0 * Programming Language: C * Projects: * Latest download count: 4618 * Maintainership: * 4 Core Developers * Last release time: Sep 2008 * Expected update: Cmockery 0.1.3 * Community(Estimated): 93 users = [wiki:Projects/GSoC/TestingReview/AceUnit AceUnit] = AceUnit is a simple unit testing framework based on CUnit and EmbUnit. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! Auto Runner ! RTEMS tests ! XML output |- | Yes | No | Yes | Yes | Support | No |- |} * License: BSD * Programming Language: C * Projects: * Download count: 13 downloads last week * Maintainership: * 2 Core Developers * Last update date: 2009-9-30 = [wiki:Projects/GSoC/TestingReview/Test-dept Test-dept] = TestDept is unit testing framework providing stubing capabilities. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | Stub | Yes | Support | No |- |} * License: GNU GPL v3 * Programming Language: C, Assembly * Projects: * Latest Download count: 231 * Maintainership: * 2 Core Developer * Last Update Date: Nov 2010 * Update Frequency before Nov 2010: every four month * Community(Estimated): 14 users = [wiki:Projects/GSoC/TestingReview/Google_Test_and_Google_Mock Google Test and Google Mock] = Google Tests is unit testing framework conforming to xUnit architecture. Google Mock is a library that can be used to provide mocking functionality to other C++ testing frameworks as well. They can be used in combination for host based testing with powerful functionalities they provide. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | Yes | No | Not support | Yes |- |} * License: New BSD license * Programming Language: C++ * Projects: * Download count: 86347 for Google Test, 30548 for Google Mock. * Maintainership: * Core Developers: 11 for Google Test, 9 for Google Mock * Community: 1655 users for Google Test, 673 for Google Mock. * Latest update date: Apr 2011 = [wiki:Projects/GSoC/TestingReview/Check Check] = Check is simple unit testing framework for test-driven development. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | No | No | Not Support | Yes |- |} * License: LGPL * Programming Language: C * Projects: * BitlBee: http://www.bitlbee.org/ * checkmk: http://micah.cowan.name/projects/checkmk/ * ctrlproxy: http://ctrlproxy.vernstok.nl/ * Daimonin: http://sourceforge.net/projects/daimonin/ * DBMail: http://www.dbmail.org/ * Enlightenment (Eet and Eina libs): http://trac.enlightenment.org/e * EXIP: http://exip.sourceforge.net/ * Expat: http://expat.sourceforge.net/ * GNOME BuildBrigade: http://live.gnome.org/BuildBrigade/ * GStreamer: http://gstreamer.freedesktop.org/ * GNUpdf: http://www.gnupdf.org/ * Iodine: http://code.kryo.se/iodine/ * Lasso: http://lasso.entrouvert.org/ * libspmt: https://svn.sable.mcgill.ca/sable/spmt/libspmt/ * Loudmouth: http://developer.imendio.com/projects/loudmouth/ * OpenSync (libopensync and libsyncml): http://www.opensync.org/ * Pigment: https://code.fluendo.com/pigment/trac/wiki * RAPP: http://savannah.nongnu.org/projects/rapp * RedStore: http://code.google.com/p/redstore/ * SCEW: http://www.nongnu.org/scew/ * SSSD: https://fedorahosted.org/sssd/ * Tinymail: http://tinymail.org/ * XCB: http://xcb.freedesktop.org/ * Download count: 1132 last week * Maintainership: * 4 Core Developers: Arien Malec, Chris Pickett, Fredrik Hugosson, and Robert Lemmen. * Last update date: 2009-9-23 = [wiki:Projects/GSoC/TestingReview/CU CU] = CU is a lightweight and simple unit testing framework suited for host-based testing with enriched set of assertions. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | No | No | Not Support | Yes |- |} * License: GNU Lesser General Public License v3 * Programming Language: C * Projects: * Maintainership: * 1 Core Developer: Daniel Fiser * Last update date: 2010-9-14 = [wiki:Projects/GSoC/TestingReview/CxxTest CxxTest] = CxxTest is a unit testing framework in C++ with xUnit architecture. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | Yes | Yes | Not Support | Yes |- |} * License: LGPL v2 * Programming Language: C++ * Projects: * Download count: 114 downloads last week * Maintainership: * Current Version: CxxTest 4.0 * 4 Core Developers * Last update date: 2012-1-7 = [wiki:Projects/GSoC/TestingReview/CuTest CuTest] = CuTest is very simple unit testing framework. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | No | No | Not Support | No |- |} * License: GPL v2 * Programming Language: C * Projects: * Download count: 56 downloads last week * Maintainership: * 1 Core Developer * Last update date: 2010-7-22 = [wiki:Projects/GSoC/TestingReview/EmbUnit EmbUnit] = EmbUnit is unit testing framework for embedded software. {| class="wikitable" |- ! xUnit ! Mock ! Test Generator ! RTEMS tests ! XML output |- | Yes | No | No | Support | Yes |- |} * License: MIT * Programming language C * Projects: * Download count: 25 last week. * Maintainership: * Core Developers: 2 * Last updated date: 2004-2-10 * A bad review on sourceforge saying that a bug has not been fixed for a long time. = Comparison Note = == Selection Criteria == The list is listed in selection priority order. In other words, CMock is considered to be the top competitive choice for this project. The priority is calculated basing on the following criteria: # Whether the framework supports both RTEMS-based and Host-based Testing # Whether the framework supports mocking functionality # The automation level of tests generation and running # Whether the framework supports xUnit architecture # The integration difficult of the framework and RTEMS application # The selection of output format of the framework # The stability of the test framework # The maintainership of the test framework # The popularity of the test framework, the number of downloads each week or in total * The frist few criteria have the higher screening weight. The following ones are also very important. The xUnit architecture support is equivalently important in the sense that frameworks supporting xUnit architecture have edges on better handling relationship between test cases than frameworks that doesn't support. == Simulation Environment == The RTEMS based tests are carried out with BSP=sis and CPU=sparc. For other CPUs and BSPs, further RTEMS based testing of converted current test suites would be simulated in future with the selected frameworks. == C++ Frameworks == Google Test and CxxUnit are testing frameworks with programming language C++, with the large set of self-produced libs and comprehensive testing functionalities, they are considered being too big to run for RTEMS based testing, with some cpus and bsps for RTEMS having only limited memory. == Mock == Although mocking capability are compared between the frameworks, no mocking functions are constructed in the RTEMS-based test simulations temporarily. == Simulation Effort == Even though all ten frameworks are compared in the above listed aspects, major simulation effort is concentrated on the top five. There is no need to simulate all of them if the test results for any top five framework is satisfiable. = =Host-based Testing==== For host-based testing framework evaluation, because host based tests are tests that does not rely on rtems APIs, I build and run the tests which are written in each framework. The screenshot are showed in each framework subpage. The screenshot is displayed only to compare the readability of the test output and amount of information contained in each framework.