TestingReview/CMock

General Comparison

CMock is a highly automated testing framework based on Unity, a unit testing framework. It have also provided handy tools supporting mocking functionality.

• Support rich sets of assertions.

• CMock supports setUp() and tearDown() functions runned before and after each test functions, which is exactly like in test fixtures. In this way, we say that CMock support xUnit.

• With the heavy usage of Ruby, which enables the high level of automation, comes with highly strict test-writing format and light tests writing effort compared to other testing frameworks supporting mocking functionality.

• Flexible. It can still be used script-free.

• It is well suited for both RTEMS based and Host based testing, where RTEMS-based testing is hard to guarantee for a large number of tests frameworks.

• It supports several compilation options, such as CMOCK_MEM_STATIC and CMOCK_MEM_SIZE, the combination of which limits the total amount of memory the testing program could use during run time. If the tests needs more memory than specified, it quit tests. Also, there are a lot of other compilation options that facilitate testing under embedded environments.

• The tests results are reported with line number and function names indicated.

• The scripts-rich feature have largely automated test running process, with test runner automatically generated and mock functions automatically generated.

• Unlike other test framework evaluated, it is upgradable to Ceedling, which is a another script driven framework based on CMock and Unity, and Ceedling is more automated than CMock.

• Failure messages can be set customarily for each assertion.

RTEMS based Tests Result

SP01 test

File:Unity sp.png?

In the above test result, because after the last test, it deletes itself, so the test output for that test does not show up. But from the total test display, it can be concluded that that test has also passed since all five tests have passed.

---

deviceio01 test

File:Unity deviceio01.png?

---

fsrofs01 test

File:Unity fsrofs01.png?

---

psx01 test

File:Unity psx01.png?

Then it prints:

Task_1: pthread_once - SUCCESSFUL (init_routine does not execute) * END OF POSIX TEST 1 *

I did not use snapshot in order to save space.

---

psxtmbarrier01 test

File:Unity psxtmbarrier01.png?

---

devfs02 test

File:Unity devfs02.png?

Host-based Test Result

Unity Output

This is the test case

File:Unity test c.png?

This is the output for this test case

File:Unity output.png?

This is the test summary

File:Unity output with summary.png?

From the above, it can be seen that it reports single assert failures as well as test case running result. In the above screenshot, the line 2011 assert fails, and that test case passes with the macro VERIFY_FAILS_END at the end. The test summary shows that all 174 test cases have passed.

In addition, with the ruby script, it can be seen that very large number of 174 test cases can be automated.

CMock Output

This compilation option in ruby script says that the total amount of memory can be used in tests is 32000 byte.

prefix: '-D' items:

• monitor
• CMOCK_MEM_STATIC
• CMOCK_MEM_SIZE = 32000

The is the test case with mocking functions:

void testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted(void) {

Executor_Init_Expect(); Executor_Run_ExpectAndReturn(TRUE); Executor_Run_ExpectAndReturn(TRUE); Executor_Run_ExpectAndReturn(TRUE); Executor_Run_ExpectAndReturn(TRUE); Executor_Run_ExpectAndReturn(FALSE);

AppMain?();

}

In the above code, Executor_Init function is being mocked and expected to execute once in the AppMain?() function. Executor_Run function is being mock with two different modes and is expected to run five times, with the top four times to return true and the fifth time to return false.

"This is the test result output:

Creating mock for Executor... gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ ../vendor/unity/src/unity.c -obuild/unity.o gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ mocks/MockExecutor.c -obuild/MockExecutor.o gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ src/Main.c -obuild/Main.o gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ ../src/cmock.c -obuild/cmock.o gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ build/TestMain_Runner.c -obuild/TestMain_Runner.o gcc -Dmonitor -DCMOCK_MEM_STATIC -D"CMOCK_MEM_SIZE = 32000" -DTEST -c -Isrc/ -I../src/ -I../vendor/unity/src/ -I../vendor/unity/examples/helper/ -Imocks/ -Itest/ test/TestMain.c -obuild/TestMain.o gcc -lm build/unity.o build/MockExecutor.o build/Main.o build/cmock.o build/TestMain_Runner.o build/TestMain.o -o build/TestMain.exe build/TestMain.exe test/TestMain.c:14:testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted:PASS ----------------------- 1 Tests 0 Failures 0 Ignored OK

It can be seen from the above process that mocking functions are created automatically according to the specification of mocking behaviors stated in the above code. The test case testMainShouldCallExecutorInitAndContinueToCallExecutorRunUntilHalted has passed.