wiki:SOCIS/2015

Version 44 (modified by Saeed Ehteshamifar, on 06/08/16 at 12:56:31) (diff)

--

European Space Agency Summer of Code in Space 2015

This page captures the students who make proposals as well as those who work on projects for RTEMS as part of ESA SOCIS 2015.

Students' Proposals

Start filling in this table for yourself as soon as possible and update as needed.

Student Completed Hello IRC Handle Proposal Title Google Docs URL
NAME Yes or No nick on #rtems Project Title Link to Google Docs for proposal (shared with mentors)
Hermann Felbinger Yes hermann_19829 Improve Code Coverage Analysis Meeting Aviation and Automotive Standards Proposal
Saeed Ehteshamifar Yes salpha RTEMS Fault tolerance: Get a fault injection tool to work with RTEMS Proposal Final Report

The columns are to be filled in as follows:

  • The Student column is for your name.
  • The Completed Hello column lets us all know whether or not you completed the require Hello World project. Based upon our experience, students who have successfully compiled and run an RTEMS application have a MUCH MUCH higher chance of success on the proposed project.
  • The IRC Handle column is your handle on IRC. RTEMS folks hang out in #rtems on freenode.net.
  • The Proposal Title should be self-explanatory.
  • The Google Docs URL is your proposal in Google Docs that can be reviewed and commented on by mentors. The proposal template should be copied and used as a baseline. This can be shared with mentors for review. Mentors can insert comments for you.

WARNING: The Google Docs version of the proposal is a WORKING copy. You MUST submit the official and final proposal using the Google Melange site. If you do not submit the final proposal via the Google Melange site, you cannot be considered

Students' Summer of Code Tracking Table

Students whose GSoC project is accepted by RTEMS shall fill in a slot with their information in the following table, which helps to centralize SoC Project Management.

Student IRC Handle Project Link Repository Link Blog
NAME nick on #rtems Link to Project Wiki page Link to project's public Github repository Link to your development blog
Hermann Felbinger hermann_19829 Wiki GitHub Blog
Saeed Ehteshamifar salpha Wiki GitHub-GRINDER GitHub-Slingshot Blog

The columns are to be filled in as follows:

  • The Student column is for your name.
  • The IRC Handle column is your handle on IRC. RTEMS folks hang out in #rtems on freenode.net.
  • The Project Link is a link to the Wiki page for your project.
  • The Repository Link is a link to the github repository for your project.
  • The Blog is a link to your blog with entries about your project. It should be updated regularly during the summer.

Student Status Updates

Each student has a section below for putting in notes from the weekly IRC meetings.

Gedare

  • May 19: Added the Status Updates page, and created this subsection as an example. For next week, I will talk to any students who missed the meeting and follow-up on any problems that were identified.

Hermann

  • May 19: Was notified that I am selected to be part of SOCIS 2015. Gedare asked me to provide some links to coverage reports after the release of RTEMS 4.11.
  • May 26: Unfortunately missed the meeting. Analyzed the QEMU emulator capabilities to check which BSPs are supported. To get familiar with the tools I am working on building the tool set for LEON2 CPU which should work with the QEMU Sparc32 emulator. To execute the test cases and run a coverage analysis I have to define a proper config file for covoar which currently only exists for pc386 BSP.
  • June 2: Still working on building and cofiguring the coverage reporting tool for LEON2. Because LEON2 is outdated I should build the tool also for LEON3.
  • June 9: Cleaned up my setup and rebuilt everything. Found an error in the framebuffer for the pc386 build. Filed a ticked explaining the configure command I used. The ticked is already closed providing a patch. Continue building the coverage reporting tool for LEON2/LEON3 now.
  • June 16: Built Qemu to emulate Sparc architectures and RTEMS for Leon2. Qemu executes the test cases and produces the traces from which the coverage reports are generated, but due to some differences in the sizes of bytes from the executable info and the symbols to analyze in static libraries configured in symbolSets.config, some errors occur in the log file when executing the sample tests.
  • June 23: Debugging the byte size mismatch of two executable test cases for some analyzed symbols. Added some debug outputs to covoar, e.g., dump executable info.
  • June 30: Working with Joel and Gedare on debugging the size mismatch for the Leon2 CPU. Fix the issue with a short term solution which does not treat the size mismatch as an error.
  • July 14: Prepared 2 blog posts explaining the setup to generate coverage reports for PC386 and LEON2 CPUs. Generated coverage report for PC386 and published it as a blog post.
  • July 28: Started working on eliminating hard coded paths from the symbol sets configuration file. Adding a commandline argument pointing to the analyzed libraries will fix this issue.
  • Aug 4: Implemented the elimination of hardcoded paths in the symbolSets.config file. I have this changes already running (the version we discussed earlier today on the devel list). I also applied these changes for the command line argument pointing to the test cases. Such that no detailed path knowledge is required when executing rtems-test script with coverage enabled.
  • Aug 11: This week I sent the patches which are required to run the coverage report generator to the devel list. Then I investigated the frontend of the couverture project, which is used to create coverage reports from execution traces as well. They support statement, decision, and MC/DC coverage analysis for their reports but only for programs written in Ada. For that they create Ada library information files which are generated by adding the -fdump-scos switch when compiling the Ada code with the GNAT compiler.

Saeed

  • May 19: Not yet admitted.
  • May 26: Admitted. Updated this page to add an entry for my proposal. Built RTEMS build tools via RSB.
  • Jun 02: Built RTEMS 4.11, gone through GRINDER, found the abstraction layer to modify (and re-implement) in order to invoke Qemu (running RTEMS). Made a repository and a blog and put the links above.
  • Jun 09: Missed meeting and did not spend time on project due to overbooked schedule for the week.
  • Jun 16: Got more knowledge about GRINDER and overall fault injection (FI) process and updated the blog accordingly. Going to model failures and tests.
  • Jun 23: Know more about Ballista approach, setup and build GRINDER, updated the blog. Try to implement runExperiment function of GRINDER in this week and get test outputs back to the DB.
  • Jun 30: Restructured the repository, implemented empty Qemu Abstraction in grinder-rtems, patched GRINDER to adapt to RTEMS. Should implement empty functions bodies now.
  • Jul 07: Extending GRINDER for RTEMS finished. Minor modifications remaining which should happen after adjustment of Slingshot to RTEMS. Repository updated with the latest changes and build instructions. Should go to Slingshot part.
  • Jul 14: Off due to preparation for my final exams.
  • Jul 21: Off due to preparation for my final exams.
  • Jul 28: Broke down task: adjusting Slingshot for RTEMS. Updated blog with steps covering adjustment process. Slingshot's test case template and compiler options should be adapted to RTEMS.
  • Aug 04: Gone through Slingshot's source code. Tried to initialize and run Slingshot on a virtual Linux instance. Fixed some issues in the process of running it. Should omit execution of test suites and port test case template to RTEMS.
  • Aug 11: Omitted execution phase of Slingshot and manually ported its test case template to RTEMS. Should adapt to RTEMS' process model with respect to how to invoke module under test's process and monitor it (catch signals).
  • Aug 18: Modified Slingshot to generate RTEMS programs. Statically running each test case by calling the corresponding test function. Should improve generated source files.
  • Aug 25: Improved code generation by using C macros and C Pre-Processor to avoid source code redundancy. Added to Slingshot a call to Linux Make to build test program. Should wrap up and prepare to publish Slingshot repository.
  • Sep 01: Wrapped up Slingshot and published it on Github. Wrote final report (which contains future works) and put the link to it on this page. End of SOCIS 2015.

Final Reports (brief)

This summer the RTEMS Project had 2 excellent students participate in the ESA Summer of Code in Space Program (SOCIS).

In the following sections, each student has summarized their individual projects.

Hermann Felbinger: Improve Code Coverage Analysis Meeting Aviation and Automotive Standards

The goal of this project is to improve the reporting of structural coverage analysis on testing RTEMS. The existing coverage analysis implementation supports statement and branch coverage. Mainly the tool for generating reports of this coverage analysis implementation is updated. The detailed coverage analyisis reports are generated providing hints to uncovered code fragments depending on the used coverage metrics. Further the coverage analysis tool is extended to evaluate also MC/DC (Modified Condition/Decision? Coverage).

In this project the planned work was to improve the coverage analysis of RTEMS. According to the ISO 26262 certification standard in automotive industry and the DO-178C for software-based avionics systems the evaluation of a complete test suite is based on different structural code coverage metrics. These metrics are statement, decision, and MC/DC. The metrics are selected on the Safety Integrity Level (SIL) (=potential for a safety hazard from A to D, where D is highest). E.g. a software application which is assigned a SIL level of D is required to achieve 100% MC/DC coverage. To use an operating system like RTEMS in safety related applications also RTEMS has to achieve 100% coverage. In this project an update of the existing coverage reporting application, which was implemented at SOCIS 2014, is provided.

I analyzed the capabilities of the instrumented QEMU emulator, called Couverture QEMU, to see for which CPU types the existing coverage analyzer and report generator can be used. Couverture QEMU generates the execution traces which are analyzed to generate coverage reports. I configured and built the existing tools for LEON2 and PC386 CPU. Running the coverage analysis for LEON2 yielded an issue caused by a mismatch of byte sizes of the execution traces and the symbols from the analyzed libraries. Together with Joel and Gedare we found and implemented a solution for this issue. Then coverage reports for PC386 and LEON2 were generated. Further hard coded paths were eliminated from configuration files pointing to the analyzed libraries and to instrumented object files (.gcno) which are produced by the gcc compiler. Then I started working on the verification of coverage results produced by gcov and a possibility to identify control statements in source code such that branches from execution traces can be mapped to conditions within a decision in the source code.

Documentation of the work in progress can be found in the blog.

Saeed Ehteshamifar: RTEMS Fault tolerance

Before this SOCIS, this project was uninitiated and had the goal of getting a fault injection tool to work with RTEMS. During this SOCIS, 3 tools were introduced to fulfill this goal:

  1. GRINDER: An open-source tool to automate test campaign execution and storing results.
  2. Slingshot: An open-source tool to generate data type-based faults for POSIX API.
  3. R: A tool to analyze results, gather statistics, extract meaningful assertions, graphs, etc.

Of these 3, the first two tools were successfully adapted to RTEMS. More specifically, more than 40 POSIX functions with more than 1300 test cases were tested using Slingshot. Results showed that 70 test cases failed which accounted for 4 functions: free, tanh, time, and times. Furthremore, CRASH failure model was proposed based on CMU's Ballista project and some theoretical preliminaries were given.

Documentation of the work in progress can be found in the blog and final report for this project which contains future works can be found here. Just to mention the next steps briefly, the followings can be listed:

  1. Integrate GRINDER’s function into Slingshot. Automate the whole procedure, plus adding an analysis tool to the chain.
  2. Test more RTEMS POSIX implementations.
  3. Add support for RTEMS Classic API.