Changes between Version 32 and Version 33 of Projects/GSoC/TestingReview


Ignore:
Timestamp:
Jul 9, 2012, 7:31:32 AM (7 years ago)
Author:
Xiaochen Pan
Comment:

Legend:

Unmodified
Added
Removed
Modified
  • Projects/GSoC/TestingReview

    v32 v33  
    11= TestingReview =
    22
     3= Comparison Note =
     4
     5== Selection Criteria ==
     6
     7
     8The list is listed in selection priority order. In other words, CMock is considered to be the top competitive choice for this project. The priority is calculated basing on the following criteria:
     9# Whether the framework supports both RTEMS-based and Host-based Testing
     10# Whether the framework supports mocking functionality
     11# The automation level of tests generation and running
     12# Whether the framework supports xUnit architecture
     13# The integration difficult of the framework and RTEMS application
     14# The selection of output format of the framework
     15# The stability of the test framework
     16# The maintainership of the test framework
     17# The popularity of the test framework, the number of downloads each week or in total
     18
     19 * The frist few criteria have the higher screening weight. The following ones are also very important. The xUnit architecture support is equivalently important in the sense that frameworks supporting xUnit architecture have edges on better handling relationship between test cases than frameworks that doesn't support.
     20== Simulation Environment ==
     21
     22
     23The RTEMS based tests are carried out with BSP=sis and CPU=sparc. For other CPUs and BSPs, further RTEMS based testing of converted current test suites would be simulated in future with the selected frameworks.
     24== C++ Frameworks ==
     25
     26
     27Google Test and CxxUnit are testing frameworks with programming language C++, with the large set of self-produced libs and comprehensive testing functionalities, they are considered being too big to run for RTEMS based testing, with some cpus and bsps for RTEMS having only limited memory.
     28== Mock ==
     29
     30
     31Although mocking capability are compared between the frameworks, no mocking functions are constructed in the RTEMS-based test simulations temporarily.
     32== Simulation Effort ==
     33
     34
     35Even though all ten frameworks are compared in the above listed aspects, major simulation effort is concentrated on the top five. There is no need to simulate all of them if the test results for any top five framework is satisfiable.
     36== Host-based Testing ==
     37
     38
     39For host-based testing framework evaluation, because host based tests are tests that does not rely on rtems APIs, I build and run the tests which are written in each framework. The screenshot are showed in each framework subpage. The screenshot is displayed only to compare the readability of the test output and amount of information contained in each framework.
     40== Converted Tests Code Link ==
     41
     42
     43This is the list of converted tests for frameworks CMock, Cmockery, Aceunit and the simulation is done RTEMS based.
     44
     45http://code.google.com/p/rtems-testing/downloads/list
    346= Unit Testing Framework Reviews =
    447
     
    284327
    285328
    286 = [wiki:Projects/GSoC/TestingReview/EmbUnit EmbUnit] =
     329= =[wiki:Projects/GSoC/TestingReview/EmbUnit EmbUnit] ==
    287330
    288331EmbUnit is unit testing framework for embedded software.
     
    310353  *  Last updated date: 2004-2-10
    311354  *  A bad review on sourceforge saying that a bug has not been fixed for a long time.
    312 = Comparison Note =
    313 
    314 == Selection Criteria ==
    315 
    316 
    317 The list is listed in selection priority order. In other words, CMock is considered to be the top competitive choice for this project. The priority is calculated basing on the following criteria:
    318 # Whether the framework supports both RTEMS-based and Host-based Testing
    319 # Whether the framework supports mocking functionality
    320 # The automation level of tests generation and running
    321 # Whether the framework supports xUnit architecture
    322 # The integration difficult of the framework and RTEMS application
    323 # The selection of output format of the framework
    324 # The stability of the test framework
    325 # The maintainership of the test framework
    326 # The popularity of the test framework, the number of downloads each week or in total
    327 
    328  * The frist few criteria have the higher screening weight. The following ones are also very important. The xUnit architecture support is equivalently important in the sense that frameworks supporting xUnit architecture have edges on better handling relationship between test cases than frameworks that doesn't support.
    329 == Simulation Environment ==
    330 
    331 
    332 The RTEMS based tests are carried out with BSP=sis and CPU=sparc. For other CPUs and BSPs, further RTEMS based testing of converted current test suites would be simulated in future with the selected frameworks.
    333 == C++ Frameworks ==
    334 
    335 
    336 Google Test and CxxUnit are testing frameworks with programming language C++, with the large set of self-produced libs and comprehensive testing functionalities, they are considered being too big to run for RTEMS based testing, with some cpus and bsps for RTEMS having only limited memory.
    337 == Mock ==
    338 
    339 
    340 Although mocking capability are compared between the frameworks, no mocking functions are constructed in the RTEMS-based test simulations temporarily.
    341 == Simulation Effort ==
    342 
    343 
    344 Even though all ten frameworks are compared in the above listed aspects, major simulation effort is concentrated on the top five. There is no need to simulate all of them if the test results for any top five framework is satisfiable.
    345 == Host-based Testing ==
    346 
    347 
    348 For host-based testing framework evaluation, because host based tests are tests that does not rely on rtems APIs, I build and run the tests which are written in each framework. The screenshot are showed in each framework subpage. The screenshot is displayed only to compare the readability of the test output and amount of information contained in each framework.
    349 =  =Converted Tests Code Link====
    350 
    351 
    352 This is the list of converted tests for frameworks CMock, Cmockery, Aceunit and the simulation is done RTEMS based.
    353 
    354 http://code.google.com/p/rtems-testing/downloads/list