wiki:Developer/Database

Version 6 (modified by Amar Takhar, on Mar 8, 2016 at 2:09:19 AM) (diff)

Progress save.

RTEMS Database

Note: The RDB has not been deployed yet but will be soon.

The RTEMS database is a central repository that holds all central project data. With this we will be able to have a fully cross-referenced repository of information that goes backwards to the creation of RTEMS to the present. It can also hold data set in the future for releases that have not happened.

Why?

Why do we want a database? The single biggest reason is V&V. We cannot do that by using a wiki there is no way to validate the data as being correct since we do not know how it relates to other data points. For example, these are questions we cannot answer or cannot easily answer:

  • How many Apps does RTEMS know?
    • What BSPs do they use?
    • What boards?
    • What CPUs?
  • How many CPU Families does RTEMS support.
  • I have CPU X does any version of RTEMS support it?
  • How reliable is RTEMS Networking? What version has the most stable IPv4 implementation
    • This will be gleaned from automatic testing we don't have to enter this!
  • What are all the changes to the RTEMS Kernel in reverse chronological order that resulted in the reliability of RTEMS going down
    • What compiler was used? What version?
  • I want to run an ARM BSP with CPU X using GCC Y with Simulator K.
    • Can I do this? Yes?
    • What version of GCC is the most reliable. What version of RTEMS?
    • What BSP is the most reliable in ARM?
    • What Boards do we test against for ARM?

The above are examples of questions that are difficult to answer or cannot be answered at all. By recording Build information from test runs and linking this to all other RTEMS data we can provide users with extremely in depth reports about a version, change, tool, platform or any other data point in the Description section below.

Let's not forget, wikis are only a snapshot of what we have now. It does not present data in a way that can be mined forward or backwards like a database can without getting difficult to maintain or hard to view.

Description

This is high level overview.

  • Architecture
    • List of RTEMS Architectures
  • BSP
  • Board
    • This is not a complete list but will have the ability for users to submit from the website including photos.
  • ChangeLog
    • Rolling list of changes that users will care about.
  • Component
    • Major components of RTEMS. API, Block Devices, Clients, Kernel, Networking, Servers, Shell...
  • CPU
    • Individual CPUs supported
  • CPU Family
    • CPU Family CPUs fall in.
  • Feature
    • A 'Feature' is a general heading such as UDP, TCP, CORBA, PPD, GDB...
  • Learn
    • Database of resources on where and how to learn using RTEMS.
  • News
    • News entries covering all the various RTEMS sites.
  • Orginisation
    • An orginisation is any org that interacts with RTEMS in some way both commercial, other projects and nonprofit. This is keyed to many other data points within the RDB.
  • Project
    • Toplevel key: What project all the resulting data in the DB relates to. This lets us clone all of this work for any projects under the RTEMS umbrella.
  • Release
    • All release versions, maturity and descriptions for RTEMS.
  • Tool
    • Tools can be Simulators, Compilers, Build tools, Doc tools...
  • Tool Type
    • Compiler, Debugger, Documentation, Memory...
  • Types
    • Availability
      • Open Source, Commercial, Dual, Restricted...
    • Architecture
      • All, 32bit, 64bit, PowerPC, Universal...
    • Platform
      • All, Windows, OS X, Unix, Linux FreeBSD...
    • Distribution
      • Source Code, Binary...
    • Change
      • General, Bugfix, Changed Added, Removed...
    • Maturity
      • Nightly, Alpha, Beta, RC, Stable, EoL...
    • AppType
      • Space, Medical, Aviation...

Uses and Explanations

The RDB is used everywhere:

  • Website
  • Test suite
  • Documentation
  • Releases
  • BuildBot

The purpose of having a database such as this is to avoid replicating data and ensure perfect resolution of our main data points. Historically RTEMS has had issues in keeping data up-to-date and accurate. This type of database removes all possibility of data being outdated as it enforces changing the data before the result can be seen.

For example, if we deprecate a release this can be set in the Release table which will automatically mark the release as deprecated in all other areas of the database. The same goes for future releases. If we set the date in the future it will fall under 'upcoming' releases until we edit the DB to set the version as released. This will make it front-and-center for the website, testing and documentation.

API

All RDB data will be available via a JSON REST API that is self-documenting. This will allow anyone who wishes to use our results and information for their own uses. API keys will not be enforced however monitoring will be done to avoid abuse

All internal tools will use this API to allow for unhooking software from our internal build farm. This will allow everything to run anywhere it wants without making changes. If someone wishes to run their own clone of our software and point it to a new API data host they can do so without any input from us.

Editing

The only way to edit the database is via the default Django admin site. It gets the job done all the focus has been put on the frontend. Eventually a nice, AJAX page will be created to ease adding ChangeLog entries as these will be the most edited item.

The admin site is self explanatory and data input is enforced with validation it is very difficult to input incorrect data.

Fixing an Issue

Since the database is relational there is only one point you will need to edit for any single error. The database has been decided to only store one data point.

When information from the database is used it will, in most cases be labeled. For instance if the description for a CPU is wrong you should look in the CPU table. The admin interface also offers a search feature that can be used to locate data within each table.

Retention

The RTEMS project has decided to retain all data collected permanently. We will never delete any recorded data this includes but is not limited to:

  • The RDB.
  • Build results.
  • Binaries resulting from a build.
    • RTEMS libraries
    • Tools (all types)
  • All source files and packages.
  • Platform images.

Validation & Verification

The purpose of retaining all data is to allow anyone to independently validate our test results using the exact same setup we used ourselves. By simulating our primary build platforms we will be able to ensure our environments are not different from what a user will use. We will be providing these images for download with tools pre-installed on a rolling basis. These images are not designed for use as a user but are strictly the result of our test building. This will make it possible to repeat results 10, 20 even 50 years down the road where it would be difficult if not impossible to collect.

All commands used to build each tool or software package will be recorded along with their build output and recorded forever.