Context switch extension is broken in SMP configurations

[Sebastian Huber]

The context switch extensions are called during _Thread_Do_dispatch():

void _Thread_Do_dispatch( Per_CPU_Control *cpu_self, ISR_Level level )
{
  Thread_Control *executing;

  executing = cpu_self->executing;

  ...
  do {
    Thread_Control *heir;

    heir = _Thread_Get_heir_and_make_it_executing( cpu_self );
    ...
    _User_extensions_Thread_switch( executing, heir );
    ...
    _Context_Switch( &executing->Registers, &heir->Registers );
    ...
  } while ( cpu_self->dispatch_necessary );
  ...
}

In uniprocessor configurations, the context switch extensions are called for all thread switches except the very first thread switch to the initialization thread. However, in SMP configurations, the context switch may be invalidated and updated in the low-level _Context_Switch() routine. See:

​https://docs.rtems.org/branches/master/c-user/symmetric_multiprocessing_services.html#thread-dispatch-details

In case such an update happens, a thread executes on the processor which was not visible to the context switch extensions. This can confuse for example event record consumers which use events generated by a context switch extension.

Fixing this is not straight forward. The context switch extensions call must move after the low-level context switch. The problem here is that we may end up in _Thread_Handler(). Adding the context switch extensions call to _Thread_Handler() covers now also the thread switch to the initialization thread. We also have to save the last executing thread of the processor. Registers or the stack cannot be used for this purpose. We have to add it to the per-processor information. Existing extensions may be affected, since now context switch extensions use the stack of the heir thread.

Calling the thread switch extensions in the low level context switch is difficult since at this point an intermediate stack is used which is only large enough to enable servicing of interrupts.

[Joel Sherrill]

As I recall, doing this in _Thread_Handler is similar to having to restore FPU. Are there other actions with this characteristic?

[Chris Johns]

Is this an issue that needs to be fixed for RTEMS 5?

[Sebastian Huber]

Replying to Chris Johns:

Is this an issue that needs to be fixed for RTEMS 5?

Yes, I think that the thread switch extensions should work in SMP configurations.

[Sebastian Huber]

Moving the thread switch extension calls into the context of the heir thread breaks for example the stack checker. It checks that the current stack pointer is in the stack area of a thread.

[Chris Johns]

Replying to Sebastian Huber:

Moving the thread switch extension calls into the context of the heir thread breaks for example the stack checker. It checks that the current stack pointer is in the stack area of a thread.

Is a patch to fix the stack checker being worked on?

I know of applications that have this running during development.

[Sebastian Huber]

Replying to Chris Johns:

Replying to Sebastian Huber:

Moving the thread switch extension calls into the context of the heir thread breaks for example the stack checker. It checks that the current stack pointer is in the stack area of a thread.

Is a patch to fix the stack checker being worked on?

I know of applications that have this running during development.

This was easy to fix:

​https://lists.rtems.org/pipermail/devel/2020-February/057759.html

I just wanted to highlight that this change can easily break existing extensions.

[Chris Johns]

Thanks.

[Sebastian Huber <sebastian.huber@…>]

In [changeset:"fcb11510c6b38a7a1f4da7205acc5461a4e18214/rtems" fcb11510/rtems]:

score: Fix context switch extensions (SMP)

In uniprocessor and SMP configurations, the context switch extensions
were called during _Thread_Do_dispatch():

void _Thread_Do_dispatch( Per_CPU_Control *cpu_self, ISR_Level level )
{

Thread_Control *executing;

executing = cpu_self->executing;

...
do {

Thread_Control *heir;

heir = _Thread_Get_heir_and_make_it_executing( cpu_self );
...
_User_extensions_Thread_switch( executing, heir );
...
_Context_Switch( &executing->Registers, &heir->Registers );
...

} while ( cpu_self->dispatch_necessary );
...

}

In uniprocessor configurations, this is fine and the context switch
extensions are called for all thread switches except the very first
thread switch to the initialization thread. However, in SMP
configurations, the context switch may be invalidated and updated in the
low-level _Context_Switch() routine. See:

​https://docs.rtems.org/branches/master/c-user/symmetric_multiprocessing_services.html#thread-dispatch-details

In case such an update happens, a thread will execute on the processor
which was not seen in the previous call of the context switch
extensions. This can confuse for example event record consumers which
use events generated by a context switch extension.

Fixing this is not straight forward. The context switch extensions call
must move after the low-level context switch. The problem here is that
we may end up in _Thread_Handler(). Adding the context switch
extensions call to _Thread_Handler() covers now also the thread switch
to the initialization thread. We also have to save the last executing
thread (ancestor) of the processor. Registers or the stack cannot be
used for this purpose. We have to add it to the per-processor
information. Existing extensions may be affected, since now context
switch extensions use the stack of the heir thread. The stack checker
is affected by this.

Calling the thread switch extensions in the low-level context switch is
difficult since at this point an intermediate stack is used which is
only large enough to enable servicing of interrupts.

Update #3885.

[Sebastian Huber <sebastian.huber@…>]

In [changeset:"198c07e5a58413bb2b16c01a15c545875d144ad5/rtems" 198c07e5/rtems]:

sptests/spextensions01: Add comment

Update #3885.

[Sebastian Huber <sebastian.huber@…>]

In [changeset:"8bd4e6a55b131a1429a082a33b48b2f56a1963be/rtems-docs" 8bd4e6a/rtems-docs]:

c-user: Document thread switch extension changes

Close #3885.