#3665 assigned enhancement

Add low level event recording infrastructure

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: normal Milestone: 5.1
Component: lib Version: 5
Severity: normal Keywords:
Cc: Blocked By:
Blocking:

Description

Add low level event recording infrastructure for system and user defined events. The infrastructure should be able to record high frequency events such as

  • SMP lock acquire/release,
  • interrupt entry/exit,
  • thread switches,
  • UMA zone allocate/free,
  • Ethernet packet input/output, etc.

It should allow post-mortem analysis in fatal error handlers, e.g. the last events should be in the record buffer, the newest event overwrites the oldest event. It should be possible to detect record buffer overflows for consumers that expect a continuous stream of events, e.g. to display the system state in real-time.

The framework should support high-end SMP machines (more than 1GHz processor frequency, more than four processors).

The existing capture engine tries to solve this problem, but its performance is not good enough for high-end production systems. The main issues are the variable-size buffers and the use of SMP locks for synchronization. To fix this, the API would change significantly.

Add a new API instead. The implementation should use per-processor data structures and no atomic read-modify-write operations. It is pretty much a per-processor ring buffer for record events.

Use the CPU counter to get the time of events. Combine it with periodic uptime events to synchronize it with CLOCK_REALTIME.

Here is an example of the

/**
 * @brief Produces a record item.
 *
 * @param event The record event without a time stamp for the item.
 * @param data The record data for the item.
 */
void rtems_record_produce( rtems_record_event event, rtems_record_data data );

function PowerPC machine code generated by GCC:

00000000 <rtems_record_produce>:
   0:   7d 00 00 a6     mfmsr   r8
   4:   7c 00 01 46     wrteei  0
   8:   7d 2e 82 a6     mfspr   r9,526
   c:   7d 50 42 a6     mfsprg  r10,0
  10:   81 4a 02 b4     lwz     r10,692(r10)
  14:   55 29 50 2a     rlwinm  r9,r9,10,0,21
  18:   7d 23 1b 78     or      r3,r9,r3
  1c:   81 2a 00 00     lwz     r9,0(r10)
  20:   80 ca 00 08     lwz     r6,8(r10)
  24:   38 e9 00 01     addi    r7,r9,1
  28:   7d 29 30 38     and     r9,r9,r6
  2c:   55 29 18 38     rlwinm  r9,r9,3,0,28
  30:   7d 2a 4a 14     add     r9,r10,r9
  34:   90 69 00 48     stw     r3,72(r9)
  38:   90 89 00 4c     stw     r4,76(r9)
  3c:   7c 20 04 ac     lwsync
  40:   90 ea 00 00     stw     r7,0(r10)
  44:   7d 00 01 06     wrtee   r8
  48:   4e 80 00 20     blr

Just 19 instructions, no branches, no stack frame, no atomic-read-modify-write, just a light weight synchronization to ensure that the consumer reads not half finished items.

Change History (7)

comment:1 Changed on Jan 29, 2019 at 12:57:00 PM by Sebastian Huber <sebastian.huber@…>

In dca6184/rtems:

Add low level event recording support

Add low level event recording infrastructure for system and user
defined events. The infrastructure is able to record high frequency
events such as

  • SMP lock acquire/release,
  • interrupt entry/exit,
  • thread switches,
  • UMA zone allocate/free, and
  • Ethernet packet input/output, etc.

It allows post-mortem analysis in fatal error handlers, e.g. the last
events are in the record buffer, the newest event overwrites the oldest
event. It is possible to detect record buffer overflows for consumers
that expect a continuous stream of events, e.g. to display the system
state in real-time.

The implementation supports high-end SMP machines (more than 1GHz
processor frequency, more than four processors).

Add a new API instead. The implementation uses per-processor data
structures and no atomic read-modify-write operations. It is uses
per-processor ring buffers to record the events.

The CPU counter is used to get the time of events. It is combined with
periodic uptime events to synchronize it with CLOCK_REALTIME.

The existing capture engine tries to solve this problem also, but its
performance is not good enough for high-end production systems. The
main issues are the variable-size buffers and the use of SMP locks for
synchronization. To fix this, the API would change significantly.

Update #3665.

comment:2 Changed on Jan 30, 2019 at 10:32:21 AM by Sebastian Huber <sebastian.huber@…>

In 03cdd5ea/rtems:

record: Add enum value for each event

Update #3665.

comment:3 Changed on Feb 1, 2019 at 8:52:23 AM by Sebastian Huber <sebastian.huber@…>

In d06b195/rtems-docs:

c-user: Add event recording configuration

Update #3665.

comment:4 Changed on Feb 1, 2019 at 8:52:25 AM by Sebastian Huber <sebastian.huber@…>

In 21c4a44/rtems-docs:

user: Add basic event recording documentation

Update #3665.

comment:5 Changed on Mar 12, 2019 at 1:03:10 PM by Sebastian Huber <sebastian.huber@…>

In d91951fb/rtems:

record: Rename internal per-CPU events

Update #3665.

comment:6 Changed on Mar 12, 2019 at 1:03:14 PM by Sebastian Huber <sebastian.huber@…>

In ebb8c28e/rtems:

record: Add system call entry/exit events

This corresponds to the Linux syscall_entry_* and syscall_exit_* events.

Update #3665.

comment:7 Changed on Mar 12, 2019 at 1:03:17 PM by Sebastian Huber <sebastian.huber@…>

In 01a5ced5/rtems:

record: Add more system events

Update #3665.

Note: See TracTickets for help on using tickets.