#2809 closed enhancement (fixed)

Reduce interrupt latency on SMP configurations during thread dispatch

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: high Milestone: 5.1
Component: score Version: 5
Severity: normal Keywords:
Cc: Blocked By:


Currently we have this situation:


"On SMP systems, scheduling decisions on one processor must be propagated to other processors through inter-processor interrupts. So, a thread dispatch which must be carried out on another processor happens not instantaneous. Thus several thread dispatch requests might be in the air and it is possible that some of them may be out of date before the corresponding processor has time to deal with them. The thread dispatch mechanism uses three per-processor variables,

  • the executing thread,
  • the heir thread, and
  • an boolean flag indicating if a thread dispatch is necessary or not.

Updates of the heir thread and the thread dispatch necessary indicator are synchronized via explicit memory barriers without the use of locks. A thread can be an heir thread on at most one processor in the system. The thread context is protected by a TTAS lock embedded in the context to ensure that it is used on at most one processor at a time. The thread post-switch actions use a per-processor lock. This implementation turned out to be quite efficient and no lock contention was observed in the test suite.

The current implementation of thread dispatching has some implications with respect to the interrupt latency. It is crucial to preserve the system invariant that a thread can execute on at most one processor in the system at a time. This is accomplished with a boolean indicator in the thread context. The processor architecture specific context switch code will mark that a thread context is no longer executing and waits that the heir context stopped execution before it restores the heir context and resumes execution of the heir thread (the boolean indicator is basically a TTAS lock). So, there is one point in time in which a processor is without a thread. This is essential to avoid cyclic dependencies in case multiple threads migrate at once. Otherwise some supervising entity is necessary to prevent deadlocks. Such a global supervisor would lead to scalability problems so this approach is not used. Currently the context switch is performed with interrupts disabled. Thus in case the heir thread is currently executing on another processor, the time of disabled interrupts is prolonged since one processor has to wait for another processor to make progress.

It is difficult to avoid this issue with the interrupt latency since interrupts normally store the context of the interrupted thread on its stack. In case a thread is marked as not executing, we must not use its thread stack to store such an interrupt context. We cannot use the heir stack before it stopped execution on another processor. If we enable interrupts during this transition, then we have to provide an alternative thread independent stack for interrupts in this time frame. This issue needs further investigation.

The problematic situation occurs in case we have a thread which executes with thread dispatching disabled and should execute on another processor (e.g. it is an heir thread on another processor). In this case the interrupts on this other processor are disabled until the thread enables thread dispatching and starts the thread dispatch sequence. The scheduler (an exception is the scheduler with thread processor affinity support) tries to avoid such a situation and checks if a new scheduled thread already executes on a processor. In case the assigned processor differs from the processor on which the thread already executes and this processor is a member of the processor set managed by this scheduler instance, it will reassign the processors to keep the already executing thread in place. Therefore normal scheduler requests will not lead to such a situation. Explicit thread migration requests, however, can lead to this situation. Explicit thread migrations may occur due to the scheduler helping protocol or explicit scheduler instance changes. The situation can also be provoked by interrupts which suspend and resume threads multiple times and produce stale asynchronous thread dispatch requests in the system."

Add an interrupt frame to the per-CPU control which can be used during context switches on SMP configurations.

Change History (15)

comment:1 Changed on Nov 18, 2016 at 7:00:46 AM by Sebastian Huber <sebastian.huber@…>

In d18560ae053857484cdb87defde44322ba67bacf/rtems:

sparc64: Rename CPU_Minimum_stack_frame

Rename SPARC64-specific CPU_Minimum_stack_frame to
SPARC64_Minimum_stack_frame. Rename SPARC64-specific

Update #2809.

comment:2 Changed on Nov 18, 2016 at 7:01:04 AM by Sebastian Huber <sebastian.huber@…>

In 427dcee8372097b0acb695d3e4645d11fccdbb6d/rtems:

sparc: Rename CPU_Minimum_stack_frame

Rename SPARC-specific CPU_Minimum_stack_frame to
SPARC_Minimum_stack_frame. Rename SPARC-specific

Update #2809.

comment:3 Changed on Nov 18, 2016 at 7:01:20 AM by Sebastian Huber <sebastian.huber@…>

In c539a865f4ffc36dcc8395a4c9b9c798e45f3eb2/rtems:

sparc: Move CPU_Interrupt_frame related defines

Move CPU_Interrupt_frame related defines to <rtems/score/cpuimpl.h>.

Update #2809.

comment:4 Changed on Nov 18, 2016 at 7:01:33 AM by Sebastian Huber <sebastian.huber@…>

In 40d592eb3e6461605838d9427dcb9f5eadc85862/rtems:

bsps/powerpc: Avoid use of CPU_Interrupt_frame

This type is not relevant for the code since only a pointer is passed

Update #2809.

comment:5 Changed on Nov 18, 2016 at 7:01:49 AM by Sebastian Huber <sebastian.huber@…>

In bf4fdb1f1dcc39635f23fbc9585140be5eedb3d4/rtems:

powerpc: Move legacy CPU_Interrupt_frame

The only remaining user of CPU_Interrupt_frame on PowerPC is the mpc5xx
support. Move it to here.

Update #2809.

comment:6 Changed on Nov 18, 2016 at 7:02:02 AM by Sebastian Huber <sebastian.huber@…>

In 2599c8e63e575ee2a6f40ca995e12d4579b8ba85/rtems:

powerpc: Add up to date CPU_Interrupt_frame

Rename ppc_exc_min_frame to CPU_Interrupt_frame. Move it and the
corresponding defines to <rtems/score/cpuimpl.h>.

Update #2809.

comment:7 Changed on Nov 18, 2016 at 7:02:16 AM by Sebastian Huber <sebastian.huber@…>

In dbeccf0ec0d34fa169a0ccaf04505f9ce4e9323b/rtems:

arm: Provide CPU_Interrupt_frame for ARMv4

Update #2809.

comment:8 Changed on Nov 18, 2016 at 7:02:42 AM by Sebastian Huber <sebastian.huber@…>

In f9aa34ddd9afa2953cf690eadb6119b1d24f4fc6/rtems:

score: Add Per_CPU_Control::Interrupt_frame

Update #2809.

comment:9 Changed on Nov 18, 2016 at 7:02:54 AM by Sebastian Huber <sebastian.huber@…>

In d5e073cde70211b2471e4366be397370e9f6ce48/rtems:

score: Allow interrupts during thread dispatch

Use a processor-specific interrupt frame during context switches in case
the executing thread is longer executes on the processor and the heir
thread is about to start execution. During this period we must not use
a thread stack for interrupt processing.

Update #2809.

comment:10 Changed on Dec 23, 2016 at 2:10:09 PM by Sebastian Huber

Priority: normalhigh

comment:11 Changed on Jan 24, 2017 at 8:00:57 AM by Sebastian Huber

Resolution: fixed
Status: newclosed

comment:12 Changed on May 11, 2017 at 7:31:02 AM by Sebastian Huber


comment:13 Changed on Oct 10, 2017 at 6:27:10 AM by Sebastian Huber

Component: SMPscore

comment:14 Changed on Oct 10, 2017 at 6:29:01 AM by Sebastian Huber

Component: scorecpukit

comment:15 Changed on Nov 9, 2017 at 6:27:14 AM by Sebastian Huber


Milestone renamed

Note: See TracTickets for help on using tickets.