source: rtems/doc/user/smp.t @ 4fa4ccb6

Last change on this file since 4fa4ccb6 was 4fa4ccb6, checked in by Sebastian Huber <sebastian.huber@…>, on Dec 16, 2015 at 5:28:48 AM

doc: SMP introduction

  • Property mode set to 100644
File size: 38.6 KB
Line 
1@c
2@c  COPYRIGHT (c) 2014.
3@c  On-Line Applications Research Corporation (OAR).
4@c  All rights reserved.
5@c
6
7@chapter Symmetric Multiprocessing Services
8
9@section Introduction
10
11The Symmetric Multiprocessing (SMP) support of the RTEMS @value{VERSION} is
12available on
13
14@itemize @bullet
15@item ARM,
16@item PowerPC, and
17@item SPARC.
18@end itemize
19
20It must be explicitly enabled via the @code{--enable-smp} configure command
21line option.  To enable SMP in the application configuration see
22@ref{Configuring a System Enable SMP Support for Applications}.  The default
23scheduler for SMP applications supports up to 32 processors and is a global
24fixed priority scheduler, see also @ref{Configuring a System Configuring
25Clustered Schedulers}.  For example applications see
26@file{testsuites/smptests}.
27
28This chapter describes the services related to Symmetric Multiprocessing
29provided by RTEMS.
30
31The application level services currently provided are:
32
33@itemize @bullet
34@item @code{rtems_get_processor_count} - Get processor count
35@item @code{rtems_get_current_processor} - Get current processor index
36@item @code{rtems_scheduler_ident} - Get ID of a scheduler
37@item @code{rtems_scheduler_get_processor_set} - Get processor set of a scheduler
38@item @code{rtems_task_get_scheduler} - Get scheduler of a task
39@item @code{rtems_task_set_scheduler} - Set scheduler of a task
40@item @code{rtems_task_get_affinity} - Get task processor affinity
41@item @code{rtems_task_set_affinity} - Set task processor affinity
42@end itemize
43
44@c
45@c
46@c
47@section Background
48
49@subsection Uniprocessor versus SMP Parallelism
50
51Uniprocessor systems have long been used in embedded systems. In this hardware
52model, there are some system execution characteristics which have long been
53taken for granted:
54
55@itemize @bullet
56@item one task executes at a time
57@item hardware events result in interrupts
58@end itemize
59
60There is no true parallelism. Even when interrupts appear to occur
61at the same time, they are processed in largely a serial fashion.
62This is true even when the interupt service routines are allowed to
63nest.  From a tasking viewpoint,  it is the responsibility of the real-time
64operatimg system to simulate parallelism by switching between tasks.
65These task switches occur in response to hardware interrupt events and explicit
66application events such as blocking for a resource or delaying.
67
68With symmetric multiprocessing, the presence of multiple processors
69allows for true concurrency and provides for cost-effective performance
70improvements. Uniprocessors tend to increase performance by increasing
71clock speed and complexity. This tends to lead to hot, power hungry
72microprocessors which are poorly suited for many embedded applications.
73
74The true concurrency is in sharp contrast to the single task and
75interrupt model of uniprocessor systems. This results in a fundamental
76change to uniprocessor system characteristics listed above. Developers
77are faced with a different set of characteristics which, in turn, break
78some existing assumptions and result in new challenges. In an SMP system
79with N processors, these are the new execution characteristics.
80
81@itemize @bullet
82@item N tasks execute in parallel
83@item hardware events result in interrupts
84@end itemize
85
86There is true parallelism with a task executing on each processor and
87the possibility of interrupts occurring on each processor. Thus in contrast
88to their being one task and one interrupt to consider on a uniprocessor,
89there are N tasks and potentially N simultaneous interrupts to consider
90on an SMP system.
91
92This increase in hardware complexity and presence of true parallelism
93results in the application developer needing to be even more cautious
94about mutual exclusion and shared data access than in a uniprocessor
95embedded system. Race conditions that never or rarely happened when an
96application executed on a uniprocessor system, become much more likely
97due to multiple threads executing in parallel. On a uniprocessor system,
98these race conditions would only happen when a task switch occurred at
99just the wrong moment. Now there are N-1 tasks executing in parallel
100all the time and this results in many more opportunities for small
101windows in critical sections to be hit.
102
103@subsection Task Affinity
104
105@cindex task affinity
106@cindex thread affinity
107
108RTEMS provides services to manipulate the affinity of a task. Affinity
109is used to specify the subset of processors in an SMP system on which
110a particular task can execute.
111
112By default, tasks have an affinity which allows them to execute on any
113available processor.
114
115Task affinity is a possible feature to be supported by SMP-aware
116schedulers. However, only a subset of the available schedulers support
117affinity. Although the behavior is scheduler specific, if the scheduler
118does not support affinity, it is likely to ignore all attempts to set
119affinity.
120
121@subsection Task Migration
122
123@cindex task migration
124@cindex thread migration
125
126With more than one processor in the system tasks can migrate from one processor
127to another.  There are three reasons why tasks migrate in RTEMS.
128
129@itemize @bullet
130@item The scheduler changes explicitly via @code{rtems_task_set_scheduler()} or
131similar directives.
132@item The task resumes execution after a blocking operation.  On a priority
133based scheduler it will evict the lowest priority task currently assigned to a
134processor in the processor set managed by the scheduler instance.
135@item The task moves temporarily to another scheduler instance due to locking
136protocols like @cite{Migratory Priority Inheritance} or the
137@cite{Multiprocessor Resource Sharing Protocol}.
138@end itemize
139
140Task migration should be avoided so that the working set of a task can stay on
141the most local cache level.
142
143The current implementation of task migration in RTEMS has some implications
144with respect to the interrupt latency.  It is crucial to preserve the system
145invariant that a task can execute on at most one processor in the system at a
146time.  This is accomplished with a boolean indicator in the task context.  The
147processor architecture specific low-level task context switch code will mark
148that a task context is no longer executing and waits that the heir context
149stopped execution before it restores the heir context and resumes execution of
150the heir task.  So there is one point in time in which a processor is without a
151task.  This is essential to avoid cyclic dependencies in case multiple tasks
152migrate at once.  Otherwise some supervising entity is necessary to prevent
153life-locks.  Such a global supervisor would lead to scalability problems so
154this approach is not used.  Currently the thread dispatch is performed with
155interrupts disabled.  So in case the heir task is currently executing on
156another processor then this prolongs the time of disabled interrupts since one
157processor has to wait for another processor to make progress.
158
159It is difficult to avoid this issue with the interrupt latency since interrupts
160normally store the context of the interrupted task on its stack.  In case a
161task is marked as not executing we must not use its task stack to store such an
162interrupt context.  We cannot use the heir stack before it stopped execution on
163another processor.  So if we enable interrupts during this transition we have
164to provide an alternative task independent stack for this time frame.  This
165issue needs further investigation.
166
167@subsection Clustered Scheduling
168
169We have clustered scheduling in case the set of processors of a system is
170partitioned into non-empty pairwise-disjoint subsets. These subsets are called
171clusters.  Clusters with a cardinality of one are partitions.  Each cluster is
172owned by exactly one scheduler instance.
173
174Clustered scheduling helps to control the worst-case latencies in
175multi-processor systems, see @cite{Brandenburg, Björn B.: Scheduling and
176Locking in Multiprocessor Real-Time Operating Systems. PhD thesis, 2011.
177@uref{http://www.cs.unc.edu/~bbb/diss/brandenburg-diss.pdf}}.  The goal is to
178reduce the amount of shared state in the system and thus prevention of lock
179contention. Modern multi-processor systems tend to have several layers of data
180and instruction caches.  With clustered scheduling it is possible to honour the
181cache topology of a system and thus avoid expensive cache synchronization
182traffic.  It is easy to implement.  The problem is to provide synchronization
183primitives for inter-cluster synchronization (more than one cluster is involved
184in the synchronization process). In RTEMS there are currently four means
185available
186
187@itemize @bullet
188@item events,
189@item message queues,
190@item semaphores using the @ref{Semaphore Manager Priority Inheritance}
191protocol (priority boosting), and
192@item semaphores using the @ref{Semaphore Manager Multiprocessor Resource
193Sharing Protocol} (MrsP).
194@end itemize
195
196The clustered scheduling approach enables separation of functions with
197real-time requirements and functions that profit from fairness and high
198throughput provided the scheduler instances are fully decoupled and adequate
199inter-cluster synchronization primitives are used.  This is work in progress.
200
201For the configuration of clustered schedulers see @ref{Configuring a System
202Configuring Clustered Schedulers}.
203
204To set the scheduler of a task see @ref{Symmetric Multiprocessing Services
205SCHEDULER_IDENT - Get ID of a scheduler} and @ref{Symmetric Multiprocessing
206Services TASK_SET_SCHEDULER - Set scheduler of a task}.
207
208@subsection Task Priority Queues
209
210Due to the support for clustered scheduling the task priority queues need
211special attention.  It makes no sense to compare the priority values of two
212different scheduler instances.  Thus, it is not possible to simply use one
213plain priority queue for tasks of different scheduler instances.
214
215One solution to this problem is to use two levels of queues.  The top level
216queue provides FIFO ordering and contains priority queues.  Each priority queue
217is associated with a scheduler instance and contains only tasks of this
218scheduler instance.  Tasks are enqueued in the priority queue corresponding to
219their scheduler instance.  In case this priority queue was empty, then it is
220appended to the FIFO.  To dequeue a task the highest priority task of the first
221priority queue in the FIFO is selected.  Then the first priority queue is
222removed from the FIFO.  In case the previously first priority queue is not
223empty, then it is appended to the FIFO.  So there is FIFO fairness with respect
224to the highest priority task of each scheduler instances. See also @cite{
225Brandenburg, Björn B.: A fully preemptive multiprocessor semaphore protocol for
226latency-sensitive real-time applications. In Proceedings of the 25th Euromicro
227Conference on Real-Time Systems (ECRTS 2013), pages 292–302, 2013.
228@uref{http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf}}.
229
230Such a two level queue may need a considerable amount of memory if fast enqueue
231and dequeue operations are desired (depends on the scheduler instance count).
232To mitigate this problem an approch of the FreeBSD kernel was implemented in
233RTEMS.  We have the invariant that a task can be enqueued on at most one task
234queue.  Thus, we need only as many queues as we have tasks.  Each task is
235equipped with spare task queue which it can give to an object on demand.  The
236task queue uses a dedicated memory space independent of the other memory used
237for the task itself. In case a task needs to block, then there are two options
238
239@itemize @bullet
240@item the object already has task queue, then the task enqueues itself to this
241already present queue and the spare task queue of the task is added to a list
242of free queues for this object, or
243@item otherwise, then the queue of the task is given to the object and the task
244enqueues itself to this queue.
245@end itemize
246
247In case the task is dequeued, then there are two options
248
249@itemize @bullet
250@item the task is the last task on the queue, then it removes this queue from
251the object and reclaims it for its own purpose, or
252@item otherwise, then the task removes one queue from the free list of the
253object and reclaims it for its own purpose.
254@end itemize
255
256Since there are usually more objects than tasks, this actually reduces the
257memory demands. In addition the objects contain only a pointer to the task
258queue structure. This helps to hide implementation details and makes it
259possible to use self-contained synchronization objects in Newlib and GCC (C++
260and OpenMP run-time support).
261
262@subsection Scheduler Helping Protocol
263
264The scheduler provides a helping protocol to support locking protocols like
265@cite{Migratory Priority Inheritance} or the @cite{Multiprocessor Resource
266Sharing Protocol}.  Each ready task can use at least one scheduler node at a
267time to gain access to a processor.  Each scheduler node has an owner, a user
268and an optional idle task.  The owner of a scheduler node is determined a task
269creation and never changes during the life time of a scheduler node.  The user
270of a scheduler node may change due to the scheduler helping protocol.  A
271scheduler node is in one of the four scheduler help states:
272
273@table @dfn
274
275@item help yourself
276
277This scheduler node is solely used by the owner task.  This task owns no
278resources using a helping protocol and thus does not take part in the scheduler
279helping protocol.  No help will be provided for other tasks.
280
281@item help active owner
282
283This scheduler node is owned by a task actively owning a resource and can be
284used to help out tasks.
285
286In case this scheduler node changes its state from ready to scheduled and the
287task executes using another node, then an idle task will be provided as a user
288of this node to temporarily execute on behalf of the owner task.  Thus lower
289priority tasks are denied access to the processors of this scheduler instance.
290
291In case a task actively owning a resource performs a blocking operation, then
292an idle task will be used also in case this node is in the scheduled state.
293
294@item help active rival
295
296This scheduler node is owned by a task actively obtaining a resource currently
297owned by another task and can be used to help out tasks.
298
299The task owning this node is ready and will give away its processor in case the
300task owning the resource asks for help.
301
302@item help passive
303
304This scheduler node is owned by a task obtaining a resource currently owned by
305another task and can be used to help out tasks.
306
307The task owning this node is blocked.
308
309@end table
310
311The following scheduler operations return a task in need for help
312
313@itemize @bullet
314@item unblock,
315@item change priority,
316@item yield, and
317@item ask for help.
318@end itemize
319
320A task in need for help is a task that encounters a scheduler state change from
321scheduled to ready (this is a pre-emption by a higher priority task) or a task
322that cannot be scheduled in an unblock operation.  Such a task can ask tasks
323which depend on resources owned by this task for help.
324
325In case it is not possible to schedule a task in need for help, then the
326scheduler nodes available for the task will be placed into the set of ready
327scheduler nodes of the corresponding scheduler instances.  Once a state change
328from ready to scheduled happens for one of scheduler nodes it will be used to
329schedule the task in need for help.
330
331The ask for help scheduler operation is used to help tasks in need for help
332returned by the operations mentioned above.  This operation is also used in
333case the root of a resource sub-tree owned by a task changes.
334
335The run-time of the ask for help procedures depend on the size of the resource
336tree of the task needing help and other resource trees in case tasks in need
337for help are produced during this operation.  Thus the worst-case latency in
338the system depends on the maximum resource tree size of the application.
339
340@subsection Critical Section Techniques and SMP
341
342As discussed earlier, SMP systems have opportunities for true parallelism
343which was not possible on uniprocessor systems. Consequently, multiple
344techniques that provided adequate critical sections on uniprocessor
345systems are unsafe on SMP systems. In this section, some of these
346unsafe techniques will be discussed.
347
348In general, applications must use proper operating system provided mutual
349exclusion mechanisms to ensure correct behavior. This primarily means
350the use of binary semaphores or mutexes to implement critical sections.
351
352@subsubsection Disable Interrupts and Interrupt Locks
353
354A low overhead means to ensure mutual exclusion in uni-processor configurations
355is to disable interrupts around a critical section.  This is commonly used in
356device driver code and throughout the operating system core.  On SMP
357configurations, however, disabling the interrupts on one processor has no
358effect on other processors.  So, this is insufficient to ensure system wide
359mutual exclusion.  The macros
360@itemize @bullet
361@item @code{rtems_interrupt_disable()},
362@item @code{rtems_interrupt_enable()}, and
363@item @code{rtems_interrupt_flush()}
364@end itemize
365are disabled on SMP configurations and its use will lead to compiler warnings
366and linker errors.  In the unlikely case that interrupts must be disabled on
367the current processor, then the
368@itemize @bullet
369@item @code{rtems_interrupt_local_disable()}, and
370@item @code{rtems_interrupt_local_enable()}
371@end itemize
372macros are now available in all configurations.
373
374Since disabling of interrupts is not enough to ensure system wide mutual
375exclusion on SMP, a new low-level synchronization primitive was added - the
376interrupt locks.  They are a simple API layer on top of the SMP locks used for
377low-level synchronization in the operating system core.  Currently they are
378implemented as a ticket lock.  On uni-processor configurations they degenerate
379to simple interrupt disable/enable sequences.  It is disallowed to acquire a
380single interrupt lock in a nested way.  This will result in an infinite loop
381with interrupts disabled.  While converting legacy code to interrupt locks care
382must be taken to avoid this situation.
383
384@example
385@group
386void legacy_code_with_interrupt_disable_enable( void )
387@{
388  rtems_interrupt_level level;
389
390  rtems_interrupt_disable( level );
391  /* Some critical stuff */
392  rtems_interrupt_enable( level );
393@}
394
395RTEMS_INTERRUPT_LOCK_DEFINE( static, lock, "Name" )
396
397void smp_ready_code_with_interrupt_lock( void )
398@{
399  rtems_interrupt_lock_context lock_context;
400
401  rtems_interrupt_lock_acquire( &lock, &lock_context );
402  /* Some critical stuff */
403  rtems_interrupt_lock_release( &lock, &lock_context );
404@}
405@end group
406@end example
407
408The @code{rtems_interrupt_lock} structure is empty on uni-processor
409configurations.  Empty structures have a different size in C
410(implementation-defined, zero in case of GCC) and C++ (implementation-defined
411non-zero value, one in case of GCC).  Thus the
412@code{RTEMS_INTERRUPT_LOCK_DECLARE()}, @code{RTEMS_INTERRUPT_LOCK_DEFINE()},
413@code{RTEMS_INTERRUPT_LOCK_MEMBER()}, and
414@code{RTEMS_INTERRUPT_LOCK_REFERENCE()} macros are provided to ensure ABI
415compatibility.
416
417@subsubsection Highest Priority Task Assumption
418
419On a uniprocessor system, it is safe to assume that when the highest
420priority task in an application executes, it will execute without being
421preempted until it voluntarily blocks. Interrupts may occur while it is
422executing, but there will be no context switch to another task unless
423the highest priority task voluntarily initiates it.
424
425Given the assumption that no other tasks will have their execution
426interleaved with the highest priority task, it is possible for this
427task to be constructed such that it does not need to acquire a binary
428semaphore or mutex for protected access to shared data.
429
430In an SMP system, it cannot be assumed there will never be a single task
431executing. It should be assumed that every processor is executing another
432application task. Further, those tasks will be ones which would not have
433been executed in a uniprocessor configuration and should be assumed to
434have data synchronization conflicts with what was formerly the highest
435priority task which executed without conflict.
436
437@subsubsection Disable Preemption
438
439On a uniprocessor system, disabling preemption in a task is very similar
440to making the highest priority task assumption. While preemption is
441disabled, no task context switches will occur unless the task initiates
442them voluntarily. And, just as with the highest priority task assumption,
443there are N-1 processors also running tasks. Thus the assumption that no
444other tasks will run while the task has preemption disabled is violated.
445
446@subsection Task Unique Data and SMP
447
448Per task variables are a service commonly provided by real-time operating
449systems for application use. They work by allowing the application
450to specify a location in memory (typically a @code{void *}) which is
451logically added to the context of a task. On each task switch, the
452location in memory is stored and each task can have a unique value in
453the same memory location. This memory location is directly accessed as a
454variable in a program.
455
456This works well in a uniprocessor environment because there is one task
457executing and one memory location containing a task-specific value. But
458it is fundamentally broken on an SMP system because there are always N
459tasks executing. With only one location in memory, N-1 tasks will not
460have the correct value.
461
462This paradigm for providing task unique data values is fundamentally
463broken on SMP systems.
464
465@subsubsection Classic API Per Task Variables
466
467The Classic API provides three directives to support per task variables. These are:
468
469@itemize @bullet
470@item @code{@value{DIRPREFIX}task_variable_add} - Associate per task variable
471@item @code{@value{DIRPREFIX}task_variable_get} - Obtain value of a a per task variable
472@item @code{@value{DIRPREFIX}task_variable_delete} - Remove per task variable
473@end itemize
474
475As task variables are unsafe for use on SMP systems, the use of these services
476must be eliminated in all software that is to be used in an SMP environment.
477The task variables API is disabled on SMP. Its use will lead to compile-time
478and link-time errors. It is recommended that the application developer consider
479the use of POSIX Keys or Thread Local Storage (TLS). POSIX Keys are available
480in all RTEMS configurations.  For the availablity of TLS on a particular
481architecture please consult the @cite{RTEMS CPU Architecture Supplement}.
482
483The only remaining user of task variables in the RTEMS code base is the Ada
484support.  So basically Ada is not available on RTEMS SMP.
485
486@subsection OpenMP
487
488OpenMP support for RTEMS is available via the GCC provided libgomp.  There is
489libgomp support for RTEMS in the POSIX configuration of libgomp since GCC 4.9
490(requires a Newlib snapshot after 2015-03-12). In GCC 6.1 or later (requires a
491Newlib snapshot after 2015-07-30 for <sys/lock.h> provided self-contained
492synchronization objects) there is a specialized libgomp configuration for RTEMS
493which offers a significantly better performance compared to the POSIX
494configuration of libgomp.  In addition application configurable thread pools
495for each scheduler instance are available in GCC 6.1 or later.
496
497The run-time configuration of libgomp is done via environment variables
498documented in the @uref{https://gcc.gnu.org/onlinedocs/libgomp/, libgomp
499manual}.  The environment variables are evaluated in a constructor function
500which executes in the context of the first initialization task before the
501actual initialization task function is called (just like a global C++
502constructor).  To set application specific values, a higher priority
503constructor function must be used to set up the environment variables.
504
505@example
506@group
507#include <stdlib.h>
508
509void __attribute__((constructor(1000))) config_libgomp( void )
510@{
511  setenv( "OMP_DISPLAY_ENV", "VERBOSE", 1 );
512  setenv( "GOMP_SPINCOUNT", "30000", 1 );
513  setenv( "GOMP_RTEMS_THREAD_POOLS", "1$2@@SCHD", 1 );
514@}
515@end group
516@end example
517
518The environment variable @env{GOMP_RTEMS_THREAD_POOLS} is RTEMS-specific.  It
519determines the thread pools for each scheduler instance.  The format for
520@env{GOMP_RTEMS_THREAD_POOLS} is a list of optional
521@code{<thread-pool-count>[$<priority>]@@<scheduler-name>} configurations
522separated by @code{:} where:
523
524@itemize @bullet
525@item @code{<thread-pool-count>} is the thread pool count for this scheduler
526instance.
527@item @code{$<priority>} is an optional priority for the worker threads of a
528thread pool according to @code{pthread_setschedparam}.  In case a priority
529value is omitted, then a worker thread will inherit the priority of the OpenMP
530master thread that created it.  The priority of the worker thread is not
531changed by libgomp after creation, even if a new OpenMP master thread using the
532worker has a different priority.
533@item @code{@@<scheduler-name>} is the scheduler instance name according to the
534RTEMS application configuration.
535@end itemize
536
537In case no thread pool configuration is specified for a scheduler instance,
538then each OpenMP master thread of this scheduler instance will use its own
539dynamically allocated thread pool.  To limit the worker thread count of the
540thread pools, each OpenMP master thread must call @code{omp_set_num_threads}.
541
542Lets suppose we have three scheduler instances @code{IO}, @code{WRK0}, and
543@code{WRK1} with @env{GOMP_RTEMS_THREAD_POOLS} set to
544@code{"1@@WRK0:3$4@@WRK1"}.  Then there are no thread pool restrictions for
545scheduler instance @code{IO}.  In the scheduler instance @code{WRK0} there is
546one thread pool available.  Since no priority is specified for this scheduler
547instance, the worker thread inherits the priority of the OpenMP master thread
548that created it.  In the scheduler instance @code{WRK1} there are three thread
549pools available and their worker threads run at priority four.
550
551@subsection Thread Dispatch Details
552
553This section gives background information to developers interested in the
554interrupt latencies introduced by thread dispatching.  A thread dispatch
555consists of all work which must be done to stop the currently executing thread
556on a processor and hand over this processor to an heir thread.
557
558On SMP systems, scheduling decisions on one processor must be propagated to
559other processors through inter-processor interrupts.  So, a thread dispatch
560which must be carried out on another processor happens not instantaneous.  Thus
561several thread dispatch requests might be in the air and it is possible that
562some of them may be out of date before the corresponding processor has time to
563deal with them.  The thread dispatch mechanism uses three per-processor
564variables,
565@itemize @bullet
566@item the executing thread,
567@item the heir thread, and
568@item an boolean flag indicating if a thread dispatch is necessary or not.
569@end itemize
570Updates of the heir thread and the thread dispatch necessary indicator are
571synchronized via explicit memory barriers without the use of locks.  A thread
572can be an heir thread on at most one processor in the system.  The thread context
573is protected by a TTAS lock embedded in the context to ensure that it is used
574on at most one processor at a time.  The thread post-switch actions use a
575per-processor lock.  This implementation turned out to be quite efficient and
576no lock contention was observed in the test suite.
577
578The current implementation of thread dispatching has some implications with
579respect to the interrupt latency.  It is crucial to preserve the system
580invariant that a thread can execute on at most one processor in the system at a
581time.  This is accomplished with a boolean indicator in the thread context.
582The processor architecture specific context switch code will mark that a thread
583context is no longer executing and waits that the heir context stopped
584execution before it restores the heir context and resumes execution of the heir
585thread (the boolean indicator is basically a TTAS lock).  So, there is one
586point in time in which a processor is without a thread.  This is essential to
587avoid cyclic dependencies in case multiple threads migrate at once.  Otherwise
588some supervising entity is necessary to prevent deadlocks.  Such a global
589supervisor would lead to scalability problems so this approach is not used.
590Currently the context switch is performed with interrupts disabled.  Thus in
591case the heir thread is currently executing on another processor, the time of
592disabled interrupts is prolonged since one processor has to wait for another
593processor to make progress.
594
595It is difficult to avoid this issue with the interrupt latency since interrupts
596normally store the context of the interrupted thread on its stack.  In case a
597thread is marked as not executing, we must not use its thread stack to store
598such an interrupt context.  We cannot use the heir stack before it stopped
599execution on another processor.  If we enable interrupts during this
600transition, then we have to provide an alternative thread independent stack for
601interrupts in this time frame.  This issue needs further investigation.
602
603The problematic situation occurs in case we have a thread which executes with
604thread dispatching disabled and should execute on another processor (e.g. it is
605an heir thread on another processor).  In this case the interrupts on this
606other processor are disabled until the thread enables thread dispatching and
607starts the thread dispatch sequence.  The scheduler (an exception is the
608scheduler with thread processor affinity support) tries to avoid such a
609situation and checks if a new scheduled thread already executes on a processor.
610In case the assigned processor differs from the processor on which the thread
611already executes and this processor is a member of the processor set managed by
612this scheduler instance, it will reassign the processors to keep the already
613executing thread in place.  Therefore normal scheduler requests will not lead
614to such a situation.  Explicit thread migration requests, however, can lead to
615this situation.  Explicit thread migrations may occur due to the scheduler
616helping protocol or explicit scheduler instance changes.  The situation can
617also be provoked by interrupts which suspend and resume threads multiple times
618and produce stale asynchronous thread dispatch requests in the system.
619
620@c
621@c
622@c
623@section Operations
624
625@subsection Setting Affinity to a Single Processor
626
627On some embedded applications targeting SMP systems, it may be beneficial to
628lock individual tasks to specific processors.  In this way, one can designate a
629processor for I/O tasks, another for computation, etc..  The following
630illustrates the code sequence necessary to assign a task an affinity for
631processor with index @code{processor_index}.
632
633@example
634@group
635#include <rtems.h>
636#include <assert.h>
637
638void pin_to_processor(rtems_id task_id, int processor_index)
639@{
640  rtems_status_code sc;
641  cpu_set_t         cpuset;
642
643  CPU_ZERO(&cpuset);
644  CPU_SET(processor_index, &cpuset);
645
646  sc = rtems_task_set_affinity(task_id, sizeof(cpuset), &cpuset);
647  assert(sc == RTEMS_SUCCESSFUL);
648@}
649@end group
650@end example
651
652It is important to note that the @code{cpuset} is not validated until the
653@code{@value{DIRPREFIX}task_set_affinity} call is made. At that point,
654it is validated against the current system configuration.
655
656@c
657@c
658@c
659@section Directives
660
661This section details the symmetric multiprocessing services.  A subsection
662is dedicated to each of these services and describes the calling sequence,
663related constants, usage, and status codes.
664
665@c
666@c rtems_get_processor_count
667@c
668@page
669@subsection GET_PROCESSOR_COUNT - Get processor count
670
671@subheading CALLING SEQUENCE:
672
673@ifset is-C
674@example
675uint32_t rtems_get_processor_count(void);
676@end example
677@end ifset
678
679@ifset is-Ada
680@end ifset
681
682@subheading DIRECTIVE STATUS CODES:
683
684The count of processors in the system.
685
686@subheading DESCRIPTION:
687
688On uni-processor configurations a value of one will be returned.
689
690On SMP configurations this returns the value of a global variable set during
691system initialization to indicate the count of utilized processors.  The
692processor count depends on the physically or virtually available processors and
693application configuration.  The value will always be less than or equal to the
694maximum count of application configured processors.
695
696@subheading NOTES:
697
698None.
699
700@c
701@c rtems_get_current_processor
702@c
703@page
704@subsection GET_CURRENT_PROCESSOR - Get current processor index
705
706@subheading CALLING SEQUENCE:
707
708@ifset is-C
709@example
710uint32_t rtems_get_current_processor(void);
711@end example
712@end ifset
713
714@ifset is-Ada
715@end ifset
716
717@subheading DIRECTIVE STATUS CODES:
718
719The index of the current processor.
720
721@subheading DESCRIPTION:
722
723On uni-processor configurations a value of zero will be returned.
724
725On SMP configurations an architecture specific method is used to obtain the
726index of the current processor in the system.  The set of processor indices is
727the range of integers starting with zero up to the processor count minus one.
728
729Outside of sections with disabled thread dispatching the current processor
730index may change after every instruction since the thread may migrate from one
731processor to another.  Sections with disabled interrupts are sections with
732thread dispatching disabled.
733
734@subheading NOTES:
735
736None.
737
738@c
739@c rtems_scheduler_ident
740@c
741@page
742@subsection SCHEDULER_IDENT - Get ID of a scheduler
743
744@subheading CALLING SEQUENCE:
745
746@ifset is-C
747@example
748rtems_status_code rtems_scheduler_ident(
749  rtems_name  name,
750  rtems_id   *id
751);
752@end example
753@end ifset
754
755@ifset is-Ada
756@end ifset
757
758@subheading DIRECTIVE STATUS CODES:
759
760@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
761@code{@value{RPREFIX}INVALID_ADDRESS} - @code{id} is NULL@*
762@code{@value{RPREFIX}INVALID_NAME} - invalid scheduler name@*
763@code{@value{RPREFIX}UNSATISFIED} - - a scheduler with this name exists, but
764the processor set of this scheduler is empty
765
766@subheading DESCRIPTION:
767
768Identifies a scheduler by its name.  The scheduler name is determined by the
769scheduler configuration.  @xref{Configuring a System Configuring Clustered
770Schedulers}.
771
772@subheading NOTES:
773
774None.
775
776@c
777@c rtems_scheduler_get_processor_set
778@c
779@page
780@subsection SCHEDULER_GET_PROCESSOR_SET - Get processor set of a scheduler
781
782@subheading CALLING SEQUENCE:
783
784@ifset is-C
785@example
786rtems_status_code rtems_scheduler_get_processor_set(
787  rtems_id   scheduler_id,
788  size_t     cpusetsize,
789  cpu_set_t *cpuset
790);
791@end example
792@end ifset
793
794@ifset is-Ada
795@end ifset
796
797@subheading DIRECTIVE STATUS CODES:
798
799@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
800@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
801@code{@value{RPREFIX}INVALID_ID} - invalid scheduler id@*
802@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
803set of processors owned by the scheduler
804
805@subheading DESCRIPTION:
806
807Returns the processor set owned by the scheduler in @code{cpuset}.  A set bit
808in the processor set means that this processor is owned by the scheduler and a
809cleared bit means the opposite.
810
811@subheading NOTES:
812
813None.
814
815@c
816@c rtems_task_get_scheduler
817@c
818@page
819@subsection TASK_GET_SCHEDULER - Get scheduler of a task
820
821@subheading CALLING SEQUENCE:
822
823@ifset is-C
824@example
825rtems_status_code rtems_task_get_scheduler(
826  rtems_id  task_id,
827  rtems_id *scheduler_id
828);
829@end example
830@end ifset
831
832@ifset is-Ada
833@end ifset
834
835@subheading DIRECTIVE STATUS CODES:
836
837@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
838@code{@value{RPREFIX}INVALID_ADDRESS} - @code{scheduler_id} is NULL@*
839@code{@value{RPREFIX}INVALID_ID} - invalid task id
840
841@subheading DESCRIPTION:
842
843Returns the scheduler identifier of a task identified by @code{task_id} in
844@code{scheduler_id}.
845
846@subheading NOTES:
847
848None.
849
850@c
851@c rtems_task_set_scheduler
852@c
853@page
854@subsection TASK_SET_SCHEDULER - Set scheduler of a task
855
856@subheading CALLING SEQUENCE:
857
858@ifset is-C
859@example
860rtems_status_code rtems_task_set_scheduler(
861  rtems_id task_id,
862  rtems_id scheduler_id
863);
864@end example
865@end ifset
866
867@ifset is-Ada
868@end ifset
869
870@subheading DIRECTIVE STATUS CODES:
871
872@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
873@code{@value{RPREFIX}INVALID_ID} - invalid task or scheduler id@*
874@code{@value{RPREFIX}INCORRECT_STATE} - the task is in the wrong state to
875perform a scheduler change
876
877@subheading DESCRIPTION:
878
879Sets the scheduler of a task identified by @code{task_id} to the scheduler
880identified by @code{scheduler_id}.  The scheduler of a task is initialized to
881the scheduler of the task that created it.
882
883@subheading NOTES:
884
885None.
886
887@subheading EXAMPLE:
888
889@example
890@group
891#include <rtems.h>
892#include <assert.h>
893
894void task(rtems_task_argument arg);
895
896void example(void)
897@{
898  rtems_status_code sc;
899  rtems_id          task_id;
900  rtems_id          scheduler_id;
901  rtems_name        scheduler_name;
902
903  scheduler_name = rtems_build_name('W', 'O', 'R', 'K');
904
905  sc = rtems_scheduler_ident(scheduler_name, &scheduler_id);
906  assert(sc == RTEMS_SUCCESSFUL);
907
908  sc = rtems_task_create(
909    rtems_build_name('T', 'A', 'S', 'K'),
910    1,
911    RTEMS_MINIMUM_STACK_SIZE,
912    RTEMS_DEFAULT_MODES,
913    RTEMS_DEFAULT_ATTRIBUTES,
914    &task_id
915  );
916  assert(sc == RTEMS_SUCCESSFUL);
917
918  sc = rtems_task_set_scheduler(task_id, scheduler_id);
919  assert(sc == RTEMS_SUCCESSFUL);
920
921  sc = rtems_task_start(task_id, task, 0);
922  assert(sc == RTEMS_SUCCESSFUL);
923@}
924@end group
925@end example
926
927@c
928@c rtems_task_get_affinity
929@c
930@page
931@subsection TASK_GET_AFFINITY - Get task processor affinity
932
933@subheading CALLING SEQUENCE:
934
935@ifset is-C
936@example
937rtems_status_code rtems_task_get_affinity(
938  rtems_id   id,
939  size_t     cpusetsize,
940  cpu_set_t *cpuset
941);
942@end example
943@end ifset
944
945@ifset is-Ada
946@end ifset
947
948@subheading DIRECTIVE STATUS CODES:
949
950@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
951@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
952@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
953@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
954the current processor affinity set of the task
955
956@subheading DESCRIPTION:
957
958Returns the current processor affinity set of the task in @code{cpuset}.  A set
959bit in the affinity set means that the task can execute on this processor and a
960cleared bit means the opposite.
961
962@subheading NOTES:
963
964None.
965
966@c
967@c rtems_task_set_affinity
968@c
969@page
970@subsection TASK_SET_AFFINITY - Set task processor affinity
971
972@subheading CALLING SEQUENCE:
973
974@ifset is-C
975@example
976rtems_status_code rtems_task_set_affinity(
977  rtems_id         id,
978  size_t           cpusetsize,
979  const cpu_set_t *cpuset
980);
981@end example
982@end ifset
983
984@ifset is-Ada
985@end ifset
986
987@subheading DIRECTIVE STATUS CODES:
988
989@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
990@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
991@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
992@code{@value{RPREFIX}INVALID_NUMBER} - invalid processor affinity set
993
994@subheading DESCRIPTION:
995
996Sets the processor affinity set for the task specified by @code{cpuset}.  A set
997bit in the affinity set means that the task can execute on this processor and a
998cleared bit means the opposite.
999
1000@subheading NOTES:
1001
1002This function will not change the scheduler of the task.  The intersection of
1003the processor affinity set and the set of processors owned by the scheduler of
1004the task must be non-empty.  It is not an error if the processor affinity set
1005contains processors that are not part of the set of processors owned by the
1006scheduler instance of the task.  A task will simply not run under normal
1007circumstances on these processors since the scheduler ignores them.  Some
1008locking protocols may temporarily use processors that are not included in the
1009processor affinity set of the task.  It is also not an error if the processor
1010affinity set contains processors that are not part of the system.
Note: See TracBrowser for help on using the repository browser.