wiki:Developer/Tracing

Version 2 (modified by Chris Johns, on Mar 15, 2015 at 5:42:17 AM) (diff)

Add CTF workflow.

Tracing

RTEMS Tracing support is an on target software based system that allows users and developers see what is happening in applications, 3rd party package or the RTEMS kernel itself in real-time.

Tracing a real-time system using just software is a complex process that involves a number of components on the host and in the target working together. Software based tracing is useful because it is portable across all architectures and board support packages how-ever tracing requires instrumenting the code, which means it is altered with the addition of code to log various events and records in real-time, and there is an execution overhead which changes the real-time profile of a system. If a user is mindful of these factors and understands the nature of the trace process these effects can be compensated for.

Trace analysis of a system can help a user find complex threading, deadlock or other related real-time issues that arise in RTEMS applications. Trace analysis can also be used by a user to audit the performance or compliance of an application to verify is it performing the desired functions.

RTEMS Trace is under development and in an open project. The RTEMS Project welcome support by users providing patches or via paid support channels that allow the work to be completed.

RTEMS Trace Components:

  1. RTEMS Trace Linker
  2. Capture Engine
  3. Common Trace Format Integration
  4. Visualisation

The RTEMS Trace Linker takes the ELF files and libraries used to build a executable and performs a link that instruments the code so the functions a user is interested in generate suitable trace records. The user provides a configuration file that defines the trace set up. The instrumented code logs the trace data using the Capture Engine. This is a thread and SMP safe module of software that takes trace records and filters the data in real-time based on trace enable records and triggers. A triggered system will log into buffers trace records that are enabled. The Capture Engine maintains a trace buffer per CPU so avoid locking overheads in SMP applications.

The Capture Engine trace buffers are transferred to the host and feed into the Babletrace tool with a suitable configuration generated by the RTEMS Trace Linker to generate CTF files. The CTF files can be viewed using tools such as Trace Compass. The RTEMS Trace Linker instruments the code with the event Ids that match the CTF configuration file so Babletrace can associate the records with the specific events.

The RTEMS Trace Linker can be configured to generate other trace output and can be used to generate custom logs for any specific purpose. For example you could create a trace log that is just an Id and timestamp so you can get a profile of the performance of you code.

RTEMS Trace Using CTF

Note: CTF support is still under development

RTEMS Trace can generate the Common Trace Format (CTF) files.

RTEMS Trace using CTF

The work flow is:

[1]
The user creates an RTEMS application in the normal manner as well as a Trace Linker configuration file. The configuration file specifies using the CTF trace mdre, the functions to trace, and the default trace enable and trigger configuration the Capture Engine uses. The user invokes the Trace Linker with the configuration and the normal link command line used to the link the application executable. The application ELF object files and libraries, including the RTEMS libraries are standard and do not need to be built specially.
[2]
The RTEMS Trace Linker reads the user's configuration file and that results in it reading the standard CTF Trace Configuration files installed with the RTEMS Trace Linker. The trace linker uses the target compiler and linker to create the trace enabled application execuable. It wraps the functions defined in the user's configuration with code that generates trace records for the Capture Engine. Each trace record is given an identifier that is used to decode the trace data. The trace wrapper code is compiled with the target compiler and the resulting ELF object file is added to the standard link command line used to link the application and the application is relinked using the wrapping option of the GNU linker.
[3]
The trace linker creates and RTEMS ELF executable that can be run on the target hardware or simulator. It also creates a CTF configuration file defining the trace records and the format of the trace records.
[4]
The user runs the application in the GNU debugger and loads the RTEMS Tools Project's Python support code. This code contains support to manage the capture engine and extract the data from the target writing it into a trace data file. The control and data extraction is not real-time, that is the application is stopped and the data extracted.
[5]
The RTEMS Trace data and the CTF configuration is passed to Babletrace where it is converted to a CTF format data file. CTF data files can be viewed with a range of visualisation tools.

Attachments (3)

Download all attachments as: .zip