wiki:Developer/Tracing

Tracing

RTEMS Tracing support is an on-target software-based system that allows users and developers see what is happening in applications, 3rd party packages, or the RTEMS kernel itself in real-time.

Tracing a real-time system using just software is a complex process that involves a number of components on the host and in the target working together. Software based tracing is useful because it is portable across all architectures and board support packages however tracing requires instrumenting the code, which means it is altered with the addition of code to log various events and records in real-time, and there is an execution overhead which changes the real-time profile of a system. If a user is mindful of these factors and understands the nature of the trace process these effects can be compensated for.

Trace analysis of a system can help a user find complex threading, deadlock or other related real-time issues that arise in RTEMS applications. Trace analysis can also be used by a user to audit the performance or compliance of an application to verify is it performing the desired functions.

RTEMS Trace is under development and is an open project. The RTEMS Project welcomes support by users providing patches or via paid support channels that allow the work to be completed by core developers.

RTEMS Trace Components:

The RTEMS Trace Linker takes the ELF files and libraries used to build a executable and performs a link that instruments the code so the functions a user is interested in generate suitable trace records. The user provides a configuration file that defines the trace set up. The instrumented code logs the trace data using the Capture Engine. This is a thread and SMP safe software module that takes trace records and filters the data in real-time based on trace enable records and triggers. A triggered system will log into buffers trace records that are enabled. The Capture Engine maintains a trace buffer per CPU to avoid locking overheads in SMP applications.

The Capture Engine trace buffers are transferred to the host and fed into the Babeltrace tool with a suitable configuration generated by the RTEMS Trace Linker to generate CTF files. The CTF files can be viewed using tools such as Trace Compass. The RTEMS Trace Linker instruments the code with the event Ids that match the CTF configuration file so Babletrace can associate the records with the specific events.

The RTEMS Trace Linker can be configured to generate other trace output and can be used to generate custom logs for any specific purpose. For example you could create a trace log that is just an Id and timestamp so you can get a profile of the performance of you code.

RTEMS Trace Using Trace Buffering

RTEMS Trace Buffering is an RTEMS Trace Linker generator that traces function entry and exit calls to a statically allocated buffer. The Trace Buffering page provide details about using tracing with trace buffering. The work flow is shown in following figure:

RTEMS Trace Buffering

[1]
The user creates an RTEMS application in the normal manner as well as a Trace Linker configuration file. The configuration file specifies using the Trace Buffering generator, the functions to trace, and the default trace enable and trigger configurations. The user invokes the Trace Linker with the configuration and the normal link command line used to the link the application executable. The application ELF object files and libraries, including the RTEMS libraries are standard and do not need to be built specially.
[2]
The RTEMS Trace Linker reads the user's configuration file and that results in it reading the standard Trace Buffering Configuration files installed with the RTEMS Trace Linker. The trace linker uses the target compiler and linker to create the trace enabled application executable. It wraps the functions defined in the user's configuration with code that captures trace records into the statically allocated buffer. The trace wrapper code is compiled with the target compiler and the resulting ELF object file is added to the standard link command line used to link the application and the application is re-linked using the wrapping option of the GNU linker.
[3]
The trace linker creates an RTEMS ELF executable that can be run on the target hardware or simulator.
[4]
The RTEMS Shell command rtrace provides commands to write the buffer to disk or print the trace records.

The Trace Buffering page details how to use RTEMS Trace Buffering.

RTEMS Trace Using CTF

Note: CTF support is still under development; ask on the lists for the current status

RTEMS Trace can generate the Common Trace Format (CTF) files. The work flow is shown in following figure:

RTEMS Trace using CTF

[1]
The user creates an RTEMS application in the normal manner as well as a Trace Linker configuration file. The configuration file specifies using the CTF trace mode, the functions to trace, and the default trace enable and trigger configuration the Capture Engine uses. The user invokes the Trace Linker with the configuration and the normal link command line used to the link the application executable. The application ELF object files and libraries, including the RTEMS libraries are standard and do not need to be built specially.
[2]
The RTEMS Trace Linker reads the user's configuration file and that results in it reading the standard CTF Trace Configuration files installed with the RTEMS Trace Linker. The trace linker uses the target compiler and linker to create the trace enabled application executable. It wraps the functions defined in the user's configuration with code that generates trace records for the Capture Engine. Each trace record is given an identifier that is used to decode the trace data. The trace wrapper code is compiled with the target compiler and the resulting ELF object file is added to the standard link command line used to link the application and the application is re-linked using the wrapping option of the GNU linker.
[3]
The trace linker creates an RTEMS ELF executable that can be run on the target hardware or simulator. It also creates a CTF configuration file defining the trace records and the format of the trace records.
[4]
The user runs the application in the GNU debugger and loads the RTEMS Tools Project's Python support code. This code contains support to manage the capture engine and extract the data from the target writing it into a trace data file. The control and data extraction is not real-time, that is the application is stopped and the data extracted.
[5]
The RTEMS Trace data and the CTF configuration is passed to Babletrace where it is converted to a CTF format data file. CTF data files can be viewed with a range of visualization tools.

RTEMS Trace Using Printk

RTEMS Trace can use the printk() call to generate console output. The work flow is shown in following figure:

RTEMS Trace using Printk

[1]
The user creates an RTEMS application in the normal manner as well as a Trace Linker configuration file. The configuration file specifies using the Printk trace mode and the functions to trace. The user invokes the Trace Linker with the configuration and the normal link command line used to the link the application executable. The application ELF object files and libraries, including the RTEMS libraries are standard and do not need to be built specially.
[2]
The RTEMS Trace Linker reads the user's configuration file and that results in it reading the standard Printk Trace Configuration files installed with the RTEMS Trace Linker. The trace linker uses the target compiler and linker to create the trace enabled application executable. It wraps the functions defined in the user's configuration with code that prints the entry with arguments and exit and return value if any. The trace wrapper code is compiled with the target compiler and the resulting ELF object file is added to the standard link command line used to link the application and the application is relinked using the wrapping option of the GNU linker.
[3]
The trace linker creates and RTEMS ELF executable that can be run on the target hardware or simulator.
[4]
The application is run in the hardware directly or using a debugger. The printk() output appears on the target console and the user can save that to a file.
Last modified 15 months ago Last modified on Jan 1, 2016, 5:54:57 PM

Attachments (3)

Download all attachments as: .zip