source: rtems/doc/user/smp.t @ 252e244

Last change on this file since 252e244 was 252e244, checked in by Sebastian Huber <sebastian.huber@…>, on Sep 4, 2015 at 11:47:30 AM

smp: Documentation

  • Property mode set to 100644
File size: 38.0 KB
Line 
1@c
2@c  COPYRIGHT (c) 2014.
3@c  On-Line Applications Research Corporation (OAR).
4@c  All rights reserved.
5@c
6
7@chapter Symmetric Multiprocessing Services
8
9@section Introduction
10
11This chapter describes the services related to Symmetric Multiprocessing
12provided by RTEMS.
13
14The application level services currently provided are:
15
16@itemize @bullet
17@item @code{rtems_get_processor_count} - Get processor count
18@item @code{rtems_get_current_processor} - Get current processor index
19@item @code{rtems_scheduler_ident} - Get ID of a scheduler
20@item @code{rtems_scheduler_get_processor_set} - Get processor set of a scheduler
21@item @code{rtems_task_get_scheduler} - Get scheduler of a task
22@item @code{rtems_task_set_scheduler} - Set scheduler of a task
23@item @code{rtems_task_get_affinity} - Get task processor affinity
24@item @code{rtems_task_set_affinity} - Set task processor affinity
25@end itemize
26
27@c
28@c
29@c
30@section Background
31
32@subsection Uniprocessor versus SMP Parallelism
33
34Uniprocessor systems have long been used in embedded systems. In this hardware
35model, there are some system execution characteristics which have long been
36taken for granted:
37
38@itemize @bullet
39@item one task executes at a time
40@item hardware events result in interrupts
41@end itemize
42
43There is no true parallelism. Even when interrupts appear to occur
44at the same time, they are processed in largely a serial fashion.
45This is true even when the interupt service routines are allowed to
46nest.  From a tasking viewpoint,  it is the responsibility of the real-time
47operatimg system to simulate parallelism by switching between tasks.
48These task switches occur in response to hardware interrupt events and explicit
49application events such as blocking for a resource or delaying.
50
51With symmetric multiprocessing, the presence of multiple processors
52allows for true concurrency and provides for cost-effective performance
53improvements. Uniprocessors tend to increase performance by increasing
54clock speed and complexity. This tends to lead to hot, power hungry
55microprocessors which are poorly suited for many embedded applications.
56
57The true concurrency is in sharp contrast to the single task and
58interrupt model of uniprocessor systems. This results in a fundamental
59change to uniprocessor system characteristics listed above. Developers
60are faced with a different set of characteristics which, in turn, break
61some existing assumptions and result in new challenges. In an SMP system
62with N processors, these are the new execution characteristics.
63
64@itemize @bullet
65@item N tasks execute in parallel
66@item hardware events result in interrupts
67@end itemize
68
69There is true parallelism with a task executing on each processor and
70the possibility of interrupts occurring on each processor. Thus in contrast
71to their being one task and one interrupt to consider on a uniprocessor,
72there are N tasks and potentially N simultaneous interrupts to consider
73on an SMP system.
74
75This increase in hardware complexity and presence of true parallelism
76results in the application developer needing to be even more cautious
77about mutual exclusion and shared data access than in a uniprocessor
78embedded system. Race conditions that never or rarely happened when an
79application executed on a uniprocessor system, become much more likely
80due to multiple threads executing in parallel. On a uniprocessor system,
81these race conditions would only happen when a task switch occurred at
82just the wrong moment. Now there are N-1 tasks executing in parallel
83all the time and this results in many more opportunities for small
84windows in critical sections to be hit.
85
86@subsection Task Affinity
87
88@cindex task affinity
89@cindex thread affinity
90
91RTEMS provides services to manipulate the affinity of a task. Affinity
92is used to specify the subset of processors in an SMP system on which
93a particular task can execute.
94
95By default, tasks have an affinity which allows them to execute on any
96available processor.
97
98Task affinity is a possible feature to be supported by SMP-aware
99schedulers. However, only a subset of the available schedulers support
100affinity. Although the behavior is scheduler specific, if the scheduler
101does not support affinity, it is likely to ignore all attempts to set
102affinity.
103
104@subsection Task Migration
105
106@cindex task migration
107@cindex thread migration
108
109With more than one processor in the system tasks can migrate from one processor
110to another.  There are three reasons why tasks migrate in RTEMS.
111
112@itemize @bullet
113@item The scheduler changes explicitly via @code{rtems_task_set_scheduler()} or
114similar directives.
115@item The task resumes execution after a blocking operation.  On a priority
116based scheduler it will evict the lowest priority task currently assigned to a
117processor in the processor set managed by the scheduler instance.
118@item The task moves temporarily to another scheduler instance due to locking
119protocols like @cite{Migratory Priority Inheritance} or the
120@cite{Multiprocessor Resource Sharing Protocol}.
121@end itemize
122
123Task migration should be avoided so that the working set of a task can stay on
124the most local cache level.
125
126The current implementation of task migration in RTEMS has some implications
127with respect to the interrupt latency.  It is crucial to preserve the system
128invariant that a task can execute on at most one processor in the system at a
129time.  This is accomplished with a boolean indicator in the task context.  The
130processor architecture specific low-level task context switch code will mark
131that a task context is no longer executing and waits that the heir context
132stopped execution before it restores the heir context and resumes execution of
133the heir task.  So there is one point in time in which a processor is without a
134task.  This is essential to avoid cyclic dependencies in case multiple tasks
135migrate at once.  Otherwise some supervising entity is necessary to prevent
136life-locks.  Such a global supervisor would lead to scalability problems so
137this approach is not used.  Currently the thread dispatch is performed with
138interrupts disabled.  So in case the heir task is currently executing on
139another processor then this prolongs the time of disabled interrupts since one
140processor has to wait for another processor to make progress.
141
142It is difficult to avoid this issue with the interrupt latency since interrupts
143normally store the context of the interrupted task on its stack.  In case a
144task is marked as not executing we must not use its task stack to store such an
145interrupt context.  We cannot use the heir stack before it stopped execution on
146another processor.  So if we enable interrupts during this transition we have
147to provide an alternative task independent stack for this time frame.  This
148issue needs further investigation.
149
150@subsection Clustered Scheduling
151
152We have clustered scheduling in case the set of processors of a system is
153partitioned into non-empty pairwise-disjoint subsets. These subsets are called
154clusters.  Clusters with a cardinality of one are partitions.  Each cluster is
155owned by exactly one scheduler instance.
156
157Clustered scheduling helps to control the worst-case latencies in
158multi-processor systems, see @cite{Brandenburg, Björn B.: Scheduling and
159Locking in Multiprocessor Real-Time Operating Systems. PhD thesis, 2011.
160@uref{http://www.cs.unc.edu/~bbb/diss/brandenburg-diss.pdf}}.  The goal is to
161reduce the amount of shared state in the system and thus prevention of lock
162contention. Modern multi-processor systems tend to have several layers of data
163and instruction caches.  With clustered scheduling it is possible to honour the
164cache topology of a system and thus avoid expensive cache synchronization
165traffic.  It is easy to implement.  The problem is to provide synchronization
166primitives for inter-cluster synchronization (more than one cluster is involved
167in the synchronization process). In RTEMS there are currently four means
168available
169
170@itemize @bullet
171@item events,
172@item message queues,
173@item semaphores using the @ref{Semaphore Manager Priority Inheritance}
174protocol (priority boosting), and
175@item semaphores using the @ref{Semaphore Manager Multiprocessor Resource
176Sharing Protocol} (MrsP).
177@end itemize
178
179The clustered scheduling approach enables separation of functions with
180real-time requirements and functions that profit from fairness and high
181throughput provided the scheduler instances are fully decoupled and adequate
182inter-cluster synchronization primitives are used.  This is work in progress.
183
184For the configuration of clustered schedulers see @ref{Configuring a System
185Configuring Clustered Schedulers}.
186
187To set the scheduler of a task see @ref{Symmetric Multiprocessing Services
188SCHEDULER_IDENT - Get ID of a scheduler} and @ref{Symmetric Multiprocessing
189Services TASK_SET_SCHEDULER - Set scheduler of a task}.
190
191@subsection Task Priority Queues
192
193Due to the support for clustered scheduling the task priority queues need
194special attention.  It makes no sense to compare the priority values of two
195different scheduler instances.  Thus, it is not possible to simply use one
196plain priority queue for tasks of different scheduler instances.
197
198One solution to this problem is to use two levels of queues.  The top level
199queue provides FIFO ordering and contains priority queues.  Each priority queue
200is associated with a scheduler instance and contains only tasks of this
201scheduler instance.  Tasks are enqueued in the priority queue corresponding to
202their scheduler instance.  In case this priority queue was empty, then it is
203appended to the FIFO.  To dequeue a task the highest priority task of the first
204priority queue in the FIFO is selected.  Then the first priority queue is
205removed from the FIFO.  In case the previously first priority queue is not
206empty, then it is appended to the FIFO.  So there is FIFO fairness with respect
207to the highest priority task of each scheduler instances. See also @cite{
208Brandenburg, Björn B.: A fully preemptive multiprocessor semaphore protocol for
209latency-sensitive real-time applications. In Proceedings of the 25th Euromicro
210Conference on Real-Time Systems (ECRTS 2013), pages 292–302, 2013.
211@uref{http://www.mpi-sws.org/~bbb/papers/pdf/ecrts13b.pdf}}.
212
213Such a two level queue may need a considerable amount of memory if fast enqueue
214and dequeue operations are desired (depends on the scheduler instance count).
215To mitigate this problem an approch of the FreeBSD kernel was implemented in
216RTEMS.  We have the invariant that a task can be enqueued on at most one task
217queue.  Thus, we need only as many queues as we have tasks.  Each task is
218equipped with spare task queue which it can give to an object on demand.  The
219task queue uses a dedicated memory space independent of the other memory used
220for the task itself. In case a task needs to block, then there are two options
221
222@itemize @bullet
223@item the object already has task queue, then the task enqueues itself to this
224already present queue and the spare task queue of the task is added to a list
225of free queues for this object, or
226@item otherwise, then the queue of the task is given to the object and the task
227enqueues itself to this queue.
228@end itemize
229
230In case the task is dequeued, then there are two options
231
232@itemize @bullet
233@item the task is the last task on the queue, then it removes this queue from
234the object and reclaims it for its own purpose, or
235@item otherwise, then the task removes one queue from the free list of the
236object and reclaims it for its own purpose.
237@end itemize
238
239Since there are usually more objects than tasks, this actually reduces the
240memory demands. In addition the objects contain only a pointer to the task
241queue structure. This helps to hide implementation details and makes it
242possible to use self-contained synchronization objects in Newlib and GCC (C++
243and OpenMP run-time support).
244
245@subsection Scheduler Helping Protocol
246
247The scheduler provides a helping protocol to support locking protocols like
248@cite{Migratory Priority Inheritance} or the @cite{Multiprocessor Resource
249Sharing Protocol}.  Each ready task can use at least one scheduler node at a
250time to gain access to a processor.  Each scheduler node has an owner, a user
251and an optional idle task.  The owner of a scheduler node is determined a task
252creation and never changes during the life time of a scheduler node.  The user
253of a scheduler node may change due to the scheduler helping protocol.  A
254scheduler node is in one of the four scheduler help states:
255
256@table @dfn
257
258@item help yourself
259
260This scheduler node is solely used by the owner task.  This task owns no
261resources using a helping protocol and thus does not take part in the scheduler
262helping protocol.  No help will be provided for other tasks.
263
264@item help active owner
265
266This scheduler node is owned by a task actively owning a resource and can be
267used to help out tasks.
268
269In case this scheduler node changes its state from ready to scheduled and the
270task executes using another node, then an idle task will be provided as a user
271of this node to temporarily execute on behalf of the owner task.  Thus lower
272priority tasks are denied access to the processors of this scheduler instance.
273
274In case a task actively owning a resource performs a blocking operation, then
275an idle task will be used also in case this node is in the scheduled state.
276
277@item help active rival
278
279This scheduler node is owned by a task actively obtaining a resource currently
280owned by another task and can be used to help out tasks.
281
282The task owning this node is ready and will give away its processor in case the
283task owning the resource asks for help.
284
285@item help passive
286
287This scheduler node is owned by a task obtaining a resource currently owned by
288another task and can be used to help out tasks.
289
290The task owning this node is blocked.
291
292@end table
293
294The following scheduler operations return a task in need for help
295
296@itemize @bullet
297@item unblock,
298@item change priority,
299@item yield, and
300@item ask for help.
301@end itemize
302
303A task in need for help is a task that encounters a scheduler state change from
304scheduled to ready (this is a pre-emption by a higher priority task) or a task
305that cannot be scheduled in an unblock operation.  Such a task can ask tasks
306which depend on resources owned by this task for help.
307
308In case it is not possible to schedule a task in need for help, then the
309scheduler nodes available for the task will be placed into the set of ready
310scheduler nodes of the corresponding scheduler instances.  Once a state change
311from ready to scheduled happens for one of scheduler nodes it will be used to
312schedule the task in need for help.
313
314The ask for help scheduler operation is used to help tasks in need for help
315returned by the operations mentioned above.  This operation is also used in
316case the root of a resource sub-tree owned by a task changes.
317
318The run-time of the ask for help procedures depend on the size of the resource
319tree of the task needing help and other resource trees in case tasks in need
320for help are produced during this operation.  Thus the worst-case latency in
321the system depends on the maximum resource tree size of the application.
322
323@subsection Critical Section Techniques and SMP
324
325As discussed earlier, SMP systems have opportunities for true parallelism
326which was not possible on uniprocessor systems. Consequently, multiple
327techniques that provided adequate critical sections on uniprocessor
328systems are unsafe on SMP systems. In this section, some of these
329unsafe techniques will be discussed.
330
331In general, applications must use proper operating system provided mutual
332exclusion mechanisms to ensure correct behavior. This primarily means
333the use of binary semaphores or mutexes to implement critical sections.
334
335@subsubsection Disable Interrupts and Interrupt Locks
336
337A low overhead means to ensure mutual exclusion in uni-processor configurations
338is to disable interrupts around a critical section.  This is commonly used in
339device driver code and throughout the operating system core.  On SMP
340configurations, however, disabling the interrupts on one processor has no
341effect on other processors.  So, this is insufficient to ensure system wide
342mutual exclusion.  The macros
343@itemize @bullet
344@item @code{rtems_interrupt_disable()},
345@item @code{rtems_interrupt_enable()}, and
346@item @code{rtems_interrupt_flush()}
347@end itemize
348are disabled on SMP configurations and its use will lead to compiler warnings
349and linker errors.  In the unlikely case that interrupts must be disabled on
350the current processor, then the
351@itemize @bullet
352@item @code{rtems_interrupt_local_disable()}, and
353@item @code{rtems_interrupt_local_enable()}
354@end itemize
355macros are now available in all configurations.
356
357Since disabling of interrupts is not enough to ensure system wide mutual
358exclusion on SMP, a new low-level synchronization primitive was added - the
359interrupt locks.  They are a simple API layer on top of the SMP locks used for
360low-level synchronization in the operating system core.  Currently they are
361implemented as a ticket lock.  On uni-processor configurations they degenerate
362to simple interrupt disable/enable sequences.  It is disallowed to acquire a
363single interrupt lock in a nested way.  This will result in an infinite loop
364with interrupts disabled.  While converting legacy code to interrupt locks care
365must be taken to avoid this situation.
366
367@example
368@group
369void legacy_code_with_interrupt_disable_enable( void )
370@{
371  rtems_interrupt_level level;
372
373  rtems_interrupt_disable( level );
374  /* Some critical stuff */
375  rtems_interrupt_enable( level );
376@}
377
378RTEMS_INTERRUPT_LOCK_DEFINE( static, lock, "Name" )
379
380void smp_ready_code_with_interrupt_lock( void )
381@{
382  rtems_interrupt_lock_context lock_context;
383
384  rtems_interrupt_lock_acquire( &lock, &lock_context );
385  /* Some critical stuff */
386  rtems_interrupt_lock_release( &lock, &lock_context );
387@}
388@end group
389@end example
390
391The @code{rtems_interrupt_lock} structure is empty on uni-processor
392configurations.  Empty structures have a different size in C
393(implementation-defined, zero in case of GCC) and C++ (implementation-defined
394non-zero value, one in case of GCC).  Thus the
395@code{RTEMS_INTERRUPT_LOCK_DECLARE()}, @code{RTEMS_INTERRUPT_LOCK_DEFINE()},
396@code{RTEMS_INTERRUPT_LOCK_MEMBER()}, and
397@code{RTEMS_INTERRUPT_LOCK_REFERENCE()} macros are provided to ensure ABI
398compatibility.
399
400@subsubsection Highest Priority Task Assumption
401
402On a uniprocessor system, it is safe to assume that when the highest
403priority task in an application executes, it will execute without being
404preempted until it voluntarily blocks. Interrupts may occur while it is
405executing, but there will be no context switch to another task unless
406the highest priority task voluntarily initiates it.
407
408Given the assumption that no other tasks will have their execution
409interleaved with the highest priority task, it is possible for this
410task to be constructed such that it does not need to acquire a binary
411semaphore or mutex for protected access to shared data.
412
413In an SMP system, it cannot be assumed there will never be a single task
414executing. It should be assumed that every processor is executing another
415application task. Further, those tasks will be ones which would not have
416been executed in a uniprocessor configuration and should be assumed to
417have data synchronization conflicts with what was formerly the highest
418priority task which executed without conflict.
419
420@subsubsection Disable Preemption
421
422On a uniprocessor system, disabling preemption in a task is very similar
423to making the highest priority task assumption. While preemption is
424disabled, no task context switches will occur unless the task initiates
425them voluntarily. And, just as with the highest priority task assumption,
426there are N-1 processors also running tasks. Thus the assumption that no
427other tasks will run while the task has preemption disabled is violated.
428
429@subsection Task Unique Data and SMP
430
431Per task variables are a service commonly provided by real-time operating
432systems for application use. They work by allowing the application
433to specify a location in memory (typically a @code{void *}) which is
434logically added to the context of a task. On each task switch, the
435location in memory is stored and each task can have a unique value in
436the same memory location. This memory location is directly accessed as a
437variable in a program.
438
439This works well in a uniprocessor environment because there is one task
440executing and one memory location containing a task-specific value. But
441it is fundamentally broken on an SMP system because there are always N
442tasks executing. With only one location in memory, N-1 tasks will not
443have the correct value.
444
445This paradigm for providing task unique data values is fundamentally
446broken on SMP systems.
447
448@subsubsection Classic API Per Task Variables
449
450The Classic API provides three directives to support per task variables. These are:
451
452@itemize @bullet
453@item @code{@value{DIRPREFIX}task_variable_add} - Associate per task variable
454@item @code{@value{DIRPREFIX}task_variable_get} - Obtain value of a a per task variable
455@item @code{@value{DIRPREFIX}task_variable_delete} - Remove per task variable
456@end itemize
457
458As task variables are unsafe for use on SMP systems, the use of these services
459must be eliminated in all software that is to be used in an SMP environment.
460The task variables API is disabled on SMP. Its use will lead to compile-time
461and link-time errors. It is recommended that the application developer consider
462the use of POSIX Keys or Thread Local Storage (TLS). POSIX Keys are available
463in all RTEMS configurations.  For the availablity of TLS on a particular
464architecture please consult the @cite{RTEMS CPU Architecture Supplement}.
465
466The only remaining user of task variables in the RTEMS code base is the Ada
467support.  So basically Ada is not available on RTEMS SMP.
468
469@subsection OpenMP
470
471OpenMP support for RTEMS is available via the GCC provided libgomp.  There is
472libgomp support for RTEMS in the POSIX configuration of libgomp since GCC 4.9
473(requires a Newlib snapshot after 2015-03-12). In GCC 6.1 or later (requires a
474Newlib snapshot after 2015-07-30 for <sys/lock.h> provided self-contained
475synchronization objects) there is a specialized libgomp configuration for RTEMS
476which offers a significantly better performance compared to the POSIX
477configuration of libgomp.  In addition application configurable thread pools
478for each scheduler instance are available in GCC 6.1 or later.
479
480The run-time configuration of libgomp is done via environment variables
481documented in the @uref{https://gcc.gnu.org/onlinedocs/libgomp/, libgomp
482manual}.  The environment variables are evaluated in a constructor function
483which executes in the context of the first initialization task before the
484actual initialization task function is called (just like a global C++
485constructor).  To set application specific values, a higher priority
486constructor function must be used to set up the environment variables.
487
488@example
489@group
490#include <stdlib.h>
491
492void __attribute__((constructor(1000))) config_libgomp( void )
493@{
494  setenv( "OMP_DISPLAY_ENV", "VERBOSE", 1 );
495  setenv( "GOMP_SPINCOUNT", "30000", 1 );
496  setenv( "GOMP_RTEMS_THREAD_POOLS", "1$2@@SCHD", 1 );
497@}
498@end group
499@end example
500
501The environment variable @env{GOMP_RTEMS_THREAD_POOLS} is RTEMS-specific.  It
502determines the thread pools for each scheduler instance.  The format for
503@env{GOMP_RTEMS_THREAD_POOLS} is a list of optional
504@code{<thread-pool-count>[$<priority>]@@<scheduler-name>} configurations
505separated by @code{:} where:
506
507@itemize @bullet
508@item @code{<thread-pool-count>} is the thread pool count for this scheduler
509instance.
510@item @code{$<priority>} is an optional priority for the worker threads of a
511thread pool according to @code{pthread_setschedparam}.  In case a priority
512value is omitted, then a worker thread will inherit the priority of the OpenMP
513master thread that created it.  The priority of the worker thread is not
514changed by libgomp after creation, even if a new OpenMP master thread using the
515worker has a different priority.
516@item @code{@@<scheduler-name>} is the scheduler instance name according to the
517RTEMS application configuration.
518@end itemize
519
520In case no thread pool configuration is specified for a scheduler instance,
521then each OpenMP master thread of this scheduler instance will use its own
522dynamically allocated thread pool.  To limit the worker thread count of the
523thread pools, each OpenMP master thread must call @code{omp_set_num_threads}.
524
525Lets suppose we have three scheduler instances @code{IO}, @code{WRK0}, and
526@code{WRK1} with @env{GOMP_RTEMS_THREAD_POOLS} set to
527@code{"1@@WRK0:3$4@@WRK1"}.  Then there are no thread pool restrictions for
528scheduler instance @code{IO}.  In the scheduler instance @code{WRK0} there is
529one thread pool available.  Since no priority is specified for this scheduler
530instance, the worker thread inherits the priority of the OpenMP master thread
531that created it.  In the scheduler instance @code{WRK1} there are three thread
532pools available and their worker threads run at priority four.
533
534@subsection Thread Dispatch Details
535
536This section gives background information to developers interested in the
537interrupt latencies introduced by thread dispatching.  A thread dispatch
538consists of all work which must be done to stop the currently executing thread
539on a processor and hand over this processor to an heir thread.
540
541On SMP systems, scheduling decisions on one processor must be propagated to
542other processors through inter-processor interrupts.  So, a thread dispatch
543which must be carried out on another processor happens not instantaneous.  Thus
544several thread dispatch requests might be in the air and it is possible that
545some of them may be out of date before the corresponding processor has time to
546deal with them.  The thread dispatch mechanism uses three per-processor
547variables,
548@itemize @bullet
549@item the executing thread,
550@item the heir thread, and
551@item an boolean flag indicating if a thread dispatch is necessary or not.
552@end itemize
553Updates of the heir thread and the thread dispatch necessary indicator are
554synchronized via explicit memory barriers without the use of locks.  A thread
555can be an heir thread on at most one processor in the system.  The thread context
556is protected by a TTAS lock embedded in the context to ensure that it is used
557on at most one processor at a time.  The thread post-switch actions use a
558per-processor lock.  This implementation turned out to be quite efficient and
559no lock contention was observed in the test suite.
560
561The current implementation of thread dispatching has some implications with
562respect to the interrupt latency.  It is crucial to preserve the system
563invariant that a thread can execute on at most one processor in the system at a
564time.  This is accomplished with a boolean indicator in the thread context.
565The processor architecture specific context switch code will mark that a thread
566context is no longer executing and waits that the heir context stopped
567execution before it restores the heir context and resumes execution of the heir
568thread (the boolean indicator is basically a TTAS lock).  So, there is one
569point in time in which a processor is without a thread.  This is essential to
570avoid cyclic dependencies in case multiple threads migrate at once.  Otherwise
571some supervising entity is necessary to prevent deadlocks.  Such a global
572supervisor would lead to scalability problems so this approach is not used.
573Currently the context switch is performed with interrupts disabled.  Thus in
574case the heir thread is currently executing on another processor, the time of
575disabled interrupts is prolonged since one processor has to wait for another
576processor to make progress.
577
578It is difficult to avoid this issue with the interrupt latency since interrupts
579normally store the context of the interrupted thread on its stack.  In case a
580thread is marked as not executing, we must not use its thread stack to store
581such an interrupt context.  We cannot use the heir stack before it stopped
582execution on another processor.  If we enable interrupts during this
583transition, then we have to provide an alternative thread independent stack for
584interrupts in this time frame.  This issue needs further investigation.
585
586The problematic situation occurs in case we have a thread which executes with
587thread dispatching disabled and should execute on another processor (e.g. it is
588an heir thread on another processor).  In this case the interrupts on this
589other processor are disabled until the thread enables thread dispatching and
590starts the thread dispatch sequence.  The scheduler (an exception is the
591scheduler with thread processor affinity support) tries to avoid such a
592situation and checks if a new scheduled thread already executes on a processor.
593In case the assigned processor differs from the processor on which the thread
594already executes and this processor is a member of the processor set managed by
595this scheduler instance, it will reassign the processors to keep the already
596executing thread in place.  Therefore normal scheduler requests will not lead
597to such a situation.  Explicit thread migration requests, however, can lead to
598this situation.  Explicit thread migrations may occur due to the scheduler
599helping protocol or explicit scheduler instance changes.  The situation can
600also be provoked by interrupts which suspend and resume threads multiple times
601and produce stale asynchronous thread dispatch requests in the system.
602
603@c
604@c
605@c
606@section Operations
607
608@subsection Setting Affinity to a Single Processor
609
610On some embedded applications targeting SMP systems, it may be beneficial to
611lock individual tasks to specific processors.  In this way, one can designate a
612processor for I/O tasks, another for computation, etc..  The following
613illustrates the code sequence necessary to assign a task an affinity for
614processor with index @code{processor_index}.
615
616@example
617@group
618#include <rtems.h>
619#include <assert.h>
620
621void pin_to_processor(rtems_id task_id, int processor_index)
622@{
623  rtems_status_code sc;
624  cpu_set_t         cpuset;
625
626  CPU_ZERO(&cpuset);
627  CPU_SET(processor_index, &cpuset);
628
629  sc = rtems_task_set_affinity(task_id, sizeof(cpuset), &cpuset);
630  assert(sc == RTEMS_SUCCESSFUL);
631@}
632@end group
633@end example
634
635It is important to note that the @code{cpuset} is not validated until the
636@code{@value{DIRPREFIX}task_set_affinity} call is made. At that point,
637it is validated against the current system configuration.
638
639@c
640@c
641@c
642@section Directives
643
644This section details the symmetric multiprocessing services.  A subsection
645is dedicated to each of these services and describes the calling sequence,
646related constants, usage, and status codes.
647
648@c
649@c rtems_get_processor_count
650@c
651@page
652@subsection GET_PROCESSOR_COUNT - Get processor count
653
654@subheading CALLING SEQUENCE:
655
656@ifset is-C
657@example
658uint32_t rtems_get_processor_count(void);
659@end example
660@end ifset
661
662@ifset is-Ada
663@end ifset
664
665@subheading DIRECTIVE STATUS CODES:
666
667The count of processors in the system.
668
669@subheading DESCRIPTION:
670
671On uni-processor configurations a value of one will be returned.
672
673On SMP configurations this returns the value of a global variable set during
674system initialization to indicate the count of utilized processors.  The
675processor count depends on the physically or virtually available processors and
676application configuration.  The value will always be less than or equal to the
677maximum count of application configured processors.
678
679@subheading NOTES:
680
681None.
682
683@c
684@c rtems_get_current_processor
685@c
686@page
687@subsection GET_CURRENT_PROCESSOR - Get current processor index
688
689@subheading CALLING SEQUENCE:
690
691@ifset is-C
692@example
693uint32_t rtems_get_current_processor(void);
694@end example
695@end ifset
696
697@ifset is-Ada
698@end ifset
699
700@subheading DIRECTIVE STATUS CODES:
701
702The index of the current processor.
703
704@subheading DESCRIPTION:
705
706On uni-processor configurations a value of zero will be returned.
707
708On SMP configurations an architecture specific method is used to obtain the
709index of the current processor in the system.  The set of processor indices is
710the range of integers starting with zero up to the processor count minus one.
711
712Outside of sections with disabled thread dispatching the current processor
713index may change after every instruction since the thread may migrate from one
714processor to another.  Sections with disabled interrupts are sections with
715thread dispatching disabled.
716
717@subheading NOTES:
718
719None.
720
721@c
722@c rtems_scheduler_ident
723@c
724@page
725@subsection SCHEDULER_IDENT - Get ID of a scheduler
726
727@subheading CALLING SEQUENCE:
728
729@ifset is-C
730@example
731rtems_status_code rtems_scheduler_ident(
732  rtems_name  name,
733  rtems_id   *id
734);
735@end example
736@end ifset
737
738@ifset is-Ada
739@end ifset
740
741@subheading DIRECTIVE STATUS CODES:
742
743@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
744@code{@value{RPREFIX}INVALID_ADDRESS} - @code{id} is NULL@*
745@code{@value{RPREFIX}INVALID_NAME} - invalid scheduler name@*
746@code{@value{RPREFIX}UNSATISFIED} - - a scheduler with this name exists, but
747the processor set of this scheduler is empty
748
749@subheading DESCRIPTION:
750
751Identifies a scheduler by its name.  The scheduler name is determined by the
752scheduler configuration.  @xref{Configuring a System Configuring Clustered
753Schedulers}.
754
755@subheading NOTES:
756
757None.
758
759@c
760@c rtems_scheduler_get_processor_set
761@c
762@page
763@subsection SCHEDULER_GET_PROCESSOR_SET - Get processor set of a scheduler
764
765@subheading CALLING SEQUENCE:
766
767@ifset is-C
768@example
769rtems_status_code rtems_scheduler_get_processor_set(
770  rtems_id   scheduler_id,
771  size_t     cpusetsize,
772  cpu_set_t *cpuset
773);
774@end example
775@end ifset
776
777@ifset is-Ada
778@end ifset
779
780@subheading DIRECTIVE STATUS CODES:
781
782@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
783@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
784@code{@value{RPREFIX}INVALID_ID} - invalid scheduler id@*
785@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
786set of processors owned by the scheduler
787
788@subheading DESCRIPTION:
789
790Returns the processor set owned by the scheduler in @code{cpuset}.  A set bit
791in the processor set means that this processor is owned by the scheduler and a
792cleared bit means the opposite.
793
794@subheading NOTES:
795
796None.
797
798@c
799@c rtems_task_get_scheduler
800@c
801@page
802@subsection TASK_GET_SCHEDULER - Get scheduler of a task
803
804@subheading CALLING SEQUENCE:
805
806@ifset is-C
807@example
808rtems_status_code rtems_task_get_scheduler(
809  rtems_id  task_id,
810  rtems_id *scheduler_id
811);
812@end example
813@end ifset
814
815@ifset is-Ada
816@end ifset
817
818@subheading DIRECTIVE STATUS CODES:
819
820@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
821@code{@value{RPREFIX}INVALID_ADDRESS} - @code{scheduler_id} is NULL@*
822@code{@value{RPREFIX}INVALID_ID} - invalid task id
823
824@subheading DESCRIPTION:
825
826Returns the scheduler identifier of a task identified by @code{task_id} in
827@code{scheduler_id}.
828
829@subheading NOTES:
830
831None.
832
833@c
834@c rtems_task_set_scheduler
835@c
836@page
837@subsection TASK_SET_SCHEDULER - Set scheduler of a task
838
839@subheading CALLING SEQUENCE:
840
841@ifset is-C
842@example
843rtems_status_code rtems_task_set_scheduler(
844  rtems_id task_id,
845  rtems_id scheduler_id
846);
847@end example
848@end ifset
849
850@ifset is-Ada
851@end ifset
852
853@subheading DIRECTIVE STATUS CODES:
854
855@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
856@code{@value{RPREFIX}INVALID_ID} - invalid task or scheduler id@*
857@code{@value{RPREFIX}INCORRECT_STATE} - the task is in the wrong state to
858perform a scheduler change
859
860@subheading DESCRIPTION:
861
862Sets the scheduler of a task identified by @code{task_id} to the scheduler
863identified by @code{scheduler_id}.  The scheduler of a task is initialized to
864the scheduler of the task that created it.
865
866@subheading NOTES:
867
868None.
869
870@subheading EXAMPLE:
871
872@example
873@group
874#include <rtems.h>
875#include <assert.h>
876
877void task(rtems_task_argument arg);
878
879void example(void)
880@{
881  rtems_status_code sc;
882  rtems_id          task_id;
883  rtems_id          scheduler_id;
884  rtems_name        scheduler_name;
885
886  scheduler_name = rtems_build_name('W', 'O', 'R', 'K');
887
888  sc = rtems_scheduler_ident(scheduler_name, &scheduler_id);
889  assert(sc == RTEMS_SUCCESSFUL);
890
891  sc = rtems_task_create(
892    rtems_build_name('T', 'A', 'S', 'K'),
893    1,
894    RTEMS_MINIMUM_STACK_SIZE,
895    RTEMS_DEFAULT_MODES,
896    RTEMS_DEFAULT_ATTRIBUTES,
897    &task_id
898  );
899  assert(sc == RTEMS_SUCCESSFUL);
900
901  sc = rtems_task_set_scheduler(task_id, scheduler_id);
902  assert(sc == RTEMS_SUCCESSFUL);
903
904  sc = rtems_task_start(task_id, task, 0);
905  assert(sc == RTEMS_SUCCESSFUL);
906@}
907@end group
908@end example
909
910@c
911@c rtems_task_get_affinity
912@c
913@page
914@subsection TASK_GET_AFFINITY - Get task processor affinity
915
916@subheading CALLING SEQUENCE:
917
918@ifset is-C
919@example
920rtems_status_code rtems_task_get_affinity(
921  rtems_id   id,
922  size_t     cpusetsize,
923  cpu_set_t *cpuset
924);
925@end example
926@end ifset
927
928@ifset is-Ada
929@end ifset
930
931@subheading DIRECTIVE STATUS CODES:
932
933@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
934@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
935@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
936@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
937the current processor affinity set of the task
938
939@subheading DESCRIPTION:
940
941Returns the current processor affinity set of the task in @code{cpuset}.  A set
942bit in the affinity set means that the task can execute on this processor and a
943cleared bit means the opposite.
944
945@subheading NOTES:
946
947None.
948
949@c
950@c rtems_task_set_affinity
951@c
952@page
953@subsection TASK_SET_AFFINITY - Set task processor affinity
954
955@subheading CALLING SEQUENCE:
956
957@ifset is-C
958@example
959rtems_status_code rtems_task_set_affinity(
960  rtems_id         id,
961  size_t           cpusetsize,
962  const cpu_set_t *cpuset
963);
964@end example
965@end ifset
966
967@ifset is-Ada
968@end ifset
969
970@subheading DIRECTIVE STATUS CODES:
971
972@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
973@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
974@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
975@code{@value{RPREFIX}INVALID_NUMBER} - invalid processor affinity set
976
977@subheading DESCRIPTION:
978
979Sets the processor affinity set for the task specified by @code{cpuset}.  A set
980bit in the affinity set means that the task can execute on this processor and a
981cleared bit means the opposite.
982
983@subheading NOTES:
984
985This function will not change the scheduler of the task.  The intersection of
986the processor affinity set and the set of processors owned by the scheduler of
987the task must be non-empty.  It is not an error if the processor affinity set
988contains processors that are not part of the set of processors owned by the
989scheduler instance of the task.  A task will simply not run under normal
990circumstances on these processors since the scheduler ignores them.  Some
991locking protocols may temporarily use processors that are not included in the
992processor affinity set of the task.  It is also not an error if the processor
993affinity set contains processors that are not part of the system.
Note: See TracBrowser for help on using the repository browser.