source: rtems/doc/user/smp.t @ 25f5730f

4.11
Last change on this file since 25f5730f was 25f5730f, checked in by Sebastian Huber <sebastian.huber@…>, on May 28, 2014 at 4:11:51 PM

score: _Scheduler_Set_affinity()

Do not change the scheduler with this function. Documentation. Coding
style.

  • Property mode set to 100644
File size: 20.3 KB
Line 
1@c
2@c  COPYRIGHT (c) 2014.
3@c  On-Line Applications Research Corporation (OAR).
4@c  All rights reserved.
5@c
6
7@chapter Symmetric Multiprocessing Services
8
9@section Introduction
10
11This chapter describes the services related to Symmetric Multiprocessing
12provided by RTEMS.
13
14The application level services currently provided are:
15
16@itemize @bullet
17@item @code{rtems_get_processor_count} - Get processor count
18@item @code{rtems_get_current_processor} - Get current processor index
19@item @code{rtems_scheduler_ident} - Get ID of a scheduler
20@item @code{rtems_scheduler_get_processor_set} - Get processor set of a scheduler
21@item @code{rtems_task_get_scheduler} - Get scheduler of a task
22@item @code{rtems_task_set_scheduler} - Set scheduler of a task
23@item @code{rtems_task_get_affinity} - Get task processor affinity
24@item @code{rtems_task_set_affinity} - Set task processor affinity
25@end itemize
26
27@c
28@c
29@c
30@section Background
31
32@subsection Uniprocessor versus SMP Parallelism
33
34Uniprocessor systems have long been used in embedded systems. In this hardware
35model, there are some system execution characteristics which have long been
36taken for granted:
37
38@itemize @bullet
39@item one task executes at a time
40@item hardware events result in interrupts
41@end itemize
42
43There is no true parallelism. Even when interrupts appear to occur
44at the same time, they are processed in largely a serial fashion.
45This is true even when the interupt service routines are allowed to
46nest.  From a tasking viewpoint,  it is the responsibility of the real-time
47operatimg system to simulate parallelism by switching between tasks.
48These task switches occur in response to hardware interrupt events and explicit
49application events such as blocking for a resource or delaying.
50
51With symmetric multiprocessing, the presence of multiple processors
52allows for true concurrency and provides for cost-effective performance
53improvements. Uniprocessors tend to increase performance by increasing
54clock speed and complexity. This tends to lead to hot, power hungry
55microprocessors which are poorly suited for many embedded applications.
56
57The true concurrency is in sharp contrast to the single task and
58interrupt model of uniprocessor systems. This results in a fundamental
59change to uniprocessor system characteristics listed above. Developers
60are faced with a different set of characteristics which, in turn, break
61some existing assumptions and result in new challenges. In an SMP system
62with N processors, these are the new execution characteristics.
63
64@itemize @bullet
65@item N tasks execute in parallel
66@item hardware events result in interrupts
67@end itemize
68
69There is true parallelism with a task executing on each processor and
70the possibility of interrupts occurring on each processor. Thus in contrast
71to their being one task and one interrupt to consider on a uniprocessor,
72there are N tasks and potentially N simultaneous interrupts to consider
73on an SMP system.
74
75This increase in hardware complexity and presence of true parallelism
76results in the application developer needing to be even more cautious
77about mutual exclusion and shared data access than in a uniprocessor
78embedded system. Race conditions that never or rarely happened when an
79application executed on a uniprocessor system, become much more likely
80due to multiple threads executing in parallel. On a uniprocessor system,
81these race conditions would only happen when a task switch occurred at
82just the wrong moment. Now there are N-1 tasks executing in parallel
83all the time and this results in many more opportunities for small
84windows in critical sections to be hit.
85
86@subsection Task Affinity
87
88@cindex task affinity
89@cindex thread affinity
90
91RTEMS provides services to manipulate the affinity of a task. Affinity
92is used to specify the subset of processors in an SMP system on which
93a particular task can execute.
94
95By default, tasks have an affinity which allows them to execute on any
96available processor.
97
98Task affinity is a possible feature to be supported by SMP-aware
99schedulers. However, only a subset of the available schedulers support
100affinity. Although the behavior is scheduler specific, if the scheduler
101does not support affinity, it is likely to ignore all attempts to set
102affinity.
103
104@subsection Task Migration
105
106@cindex task migration
107@cindex thread migration
108
109With more than one processor in the system tasks can migrate from one processor
110to another.  There are three reasons why tasks migrate in RTEMS.
111
112@itemize @bullet
113@item The scheduler changes explicitly via @code{rtems_task_set_scheduler()} or
114similar directives.
115@item The task resumes execution after a blocking operation.  On a priority
116based scheduler it will evict the lowest priority task currently assigned to a
117processor in the processor set managed by the scheduler instance.
118@item The task moves temporarily to another scheduler instance due to locking
119protocols like @cite{Migratory Priority Inheritance} or the
120@cite{Multiprocessor Resource Sharing Protocol}.
121@end itemize
122
123Task migration should be avoided so that the working set of a task can stay on
124the most local cache level.
125
126The current implementation of task migration in RTEMS has some implications
127with respect to the interrupt latency.  It is crucial to preserve the system
128invariant that a task can execute on at most one processor in the system at a
129time.  This is accomplished with a boolean indicator in the task context.  The
130processor architecture specific low-level task context switch code will mark
131that a task context is no longer executing and waits that the heir context
132stopped execution before it restores the heir context and resumes execution of
133the heir task.  So there is one point in time in which a processor is without a
134task.  This is essential to avoid cyclic dependencies in case multiple tasks
135migrate at once.  Otherwise some supervising entity is necessary to prevent
136life-locks.  Such a global supervisor would lead to scalability problems so
137this approach is not used.  Currently the thread dispatch is performed with
138interrupts disabled.  So in case the heir task is currently executing on
139another processor then this prolongs the time of disabled interrupts since one
140processor has to wait for another processor to make progress.
141
142It is difficult to avoid this issue with the interrupt latency since interrupts
143normally store the context of the interrupted task on its stack.  In case a
144task is marked as not executing we must not use its task stack to store such an
145interrupt context.  We cannot use the heir stack before it stopped execution on
146another processor.  So if we enable interrupts during this transition we have
147to provide an alternative task independent stack for this time frame.  This
148issue needs further investigation.
149
150@subsection Critical Section Techniques and SMP
151
152As discussed earlier, SMP systems have opportunities for true parallelism
153which was not possible on uniprocessor systems. Consequently, multiple
154techniques that provided adequate critical sections on uniprocessor
155systems are unsafe on SMP systems. In this section, some of these
156unsafe techniques will be discussed.
157
158In general, applications must use proper operating system provided mutual
159exclusion mechanisms to ensure correct behavior. This primarily means
160the use of binary semaphores or mutexes to implement critical sections.
161
162@subsubsection Disable Interrupts
163
164Again on a uniprocessor system, there is only a single processor which
165logically executes a single task and takes interrupts. On an SMP system,
166each processor may take an interrupt. When the application disables
167interrupts, it generally does so by altering a processor register to
168mask interrupts and later to re-enable them. On a uniprocessor system,
169changing this in the single processor is sufficient. However, on an SMP
170system, this register in @strong{ALL} processors must be changed. There
171are no comparable capabilities in an SMP system to disable all interrupts
172across all processors.
173
174@subsubsection Highest Priority Task Assumption
175
176On a uniprocessor system, it is safe to assume that when the highest
177priority task in an application executes, it will execute without being
178preempted until it voluntarily blocks. Interrupts may occur while it is
179executing, but there will be no context switch to another task unless
180the highest priority task voluntarily initiates it.
181
182Given the assumption that no other tasks will have their execution
183interleaved with the highest priority task, it is possible for this
184task to be constructed such that it does not need to acquire a binary
185semaphore or mutex for protected access to shared data.
186
187In an SMP system, it cannot be assumed there will never be a single task
188executing. It should be assumed that every processor is executing another
189application task. Further, those tasks will be ones which would not have
190been executed in a uniprocessor configuration and should be assumed to
191have data synchronization conflicts with what was formerly the highest
192priority task which executed without conflict.
193
194@subsubsection Disable Preemption
195
196On a uniprocessor system, disabling preemption in a task is very similar
197to making the highest priority task assumption. While preemption is
198disabled, no task context switches will occur unless the task initiates
199them voluntarily. And, just as with the highest priority task assumption,
200there are N-1 processors also running tasks. Thus the assumption that no
201other tasks will run while the task has preemption disabled is violated.
202
203@subsection Task Unique Data and SMP
204
205Per task variables are a service commonly provided by real-time operating
206systems for application use. They work by allowing the application
207to specify a location in memory (typically a @code{void *}) which is
208logically added to the context of a task. On each task switch, the
209location in memory is stored and each task can have a unique value in
210the same memory location. This memory location is directly accessed as a
211variable in a program.
212
213This works well in a uniprocessor environment because there is one task
214executing and one memory location containing a task-specific value. But
215it is fundamentally broken on an SMP system because there are always N
216tasks executing. With only one location in memory, N-1 tasks will not
217have the correct value.
218
219This paradigm for providing task unique data values is fundamentally
220broken on SMP systems.
221
222@subsubsection Classic API Per Task Variables
223
224The Classic API provides three directives to support per task variables. These are:
225
226@itemize @bullet
227@item @code{@value{DIRPREFIX}task_variable_add} - Associate per task variable
228@item @code{@value{DIRPREFIX}task_variable_get} - Obtain value of a a per task variable
229@item @code{@value{DIRPREFIX}task_variable_delete} - Remove per task variable
230@end itemize
231
232As task variables are unsafe for use on SMP systems, the use of these
233services should be eliminated in all software that is to be used in
234an SMP environment. It is recommended that the application developer
235consider the use of POSIX Keys or Thread Local Storage (TLS). POSIX Keys
236are not enabled in all RTEMS configurations.
237
238@b{STATUS}: As of March 2014, some support services in the
239@code{rtems/cpukit} use per task variables. When these uses are
240eliminated, the per task variable directives will be disabled when
241building RTEMS in SMP configuration.
242
243@c
244@c
245@c
246@section Operations
247
248@subsection Setting Affinity to a Single Processor
249
250On some embedded applications targeting SMP systems, it may be beneficial to
251lock individual tasks to specific processors.  In this way, one can designate a
252processor for I/O tasks, another for computation, etc..  The following
253illustrates the code sequence necessary to assign a task an affinity for
254processor with index @code{processor_index}.
255
256@example
257@group
258#include <rtems.h>
259#include <assert.h>
260
261void pin_to_processor(rtems_id task_id, int processor_index)
262@{
263  rtems_status_code sc;
264  cpu_set_t         cpuset;
265
266  CPU_ZERO(&cpuset);
267  CPU_SET(processor_index, &cpuset);
268
269  sc = rtems_task_set_affinity(task_id, sizeof(cpuset), &cpuset);
270  assert(sc == RTEMS_SUCCESSFUL);
271@}
272@end group
273@end example
274
275It is important to note that the @code{cpuset} is not validated until the
276@code{@value{DIRPREFIX}task_set_affinity} call is made. At that point,
277it is validated against the current system configuration.
278
279@c
280@c
281@c
282@section Directives
283
284This section details the symmetric multiprocessing services.  A subsection
285is dedicated to each of these services and describes the calling sequence,
286related constants, usage, and status codes.
287
288@c
289@c rtems_get_processor_count
290@c
291@page
292@subsection GET_PROCESSOR_COUNT - Get processor count
293
294@subheading CALLING SEQUENCE:
295
296@ifset is-C
297@example
298uint32_t rtems_get_processor_count(void);
299@end example
300@end ifset
301
302@ifset is-Ada
303@end ifset
304
305@subheading DIRECTIVE STATUS CODES:
306
307The count of processors in the system.
308
309@subheading DESCRIPTION:
310
311On uni-processor configurations a value of one will be returned.
312
313On SMP configurations this returns the value of a global variable set during
314system initialization to indicate the count of utilized processors.  The
315processor count depends on the physically or virtually available processors and
316application configuration.  The value will always be less than or equal to the
317maximum count of application configured processors.
318
319@subheading NOTES:
320
321None.
322
323@c
324@c rtems_get_current_processor
325@c
326@page
327@subsection GET_CURRENT_PROCESSOR - Get current processor index
328
329@subheading CALLING SEQUENCE:
330
331@ifset is-C
332@example
333uint32_t rtems_get_current_processor(void);
334@end example
335@end ifset
336
337@ifset is-Ada
338@end ifset
339
340@subheading DIRECTIVE STATUS CODES:
341
342The index of the current processor.
343
344@subheading DESCRIPTION:
345
346On uni-processor configurations a value of zero will be returned.
347
348On SMP configurations an architecture specific method is used to obtain the
349index of the current processor in the system.  The set of processor indices is
350the range of integers starting with zero up to the processor count minus one.
351
352Outside of sections with disabled thread dispatching the current processor
353index may change after every instruction since the thread may migrate from one
354processor to another.  Sections with disabled interrupts are sections with
355thread dispatching disabled.
356
357@subheading NOTES:
358
359None.
360
361@c
362@c rtems_scheduler_ident
363@c
364@page
365@subsection SCHEDULER_IDENT - Get ID of a scheduler
366
367@subheading CALLING SEQUENCE:
368
369@ifset is-C
370@example
371rtems_status_code rtems_scheduler_ident(
372  rtems_name  name,
373  rtems_id   *id
374);
375@end example
376@end ifset
377
378@ifset is-Ada
379@end ifset
380
381@subheading DIRECTIVE STATUS CODES:
382
383@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
384@code{@value{RPREFIX}INVALID_ADDRESS} - @code{id} is NULL@*
385@code{@value{RPREFIX}INVALID_NAME} - invalid scheduler name@*
386@code{@value{RPREFIX}UNSATISFIED} - - a scheduler with this name exists, but
387the processor set of this scheduler is empty
388
389@subheading DESCRIPTION:
390
391Identifies a scheduler by its name.  The scheduler name is determined by the
392scheduler configuration.  @xref{Configuring a System Configuring
393Clustered/Partitioned Schedulers}.
394
395@subheading NOTES:
396
397None.
398
399@c
400@c rtems_scheduler_get_processor_set
401@c
402@page
403@subsection SCHEDULER_GET_PROCESSOR_SET - Get processor set of a scheduler
404
405@subheading CALLING SEQUENCE:
406
407@ifset is-C
408@example
409rtems_status_code rtems_scheduler_get_processor_set(
410  rtems_id   scheduler_id,
411  size_t     cpusetsize,
412  cpu_set_t *cpuset
413);
414@end example
415@end ifset
416
417@ifset is-Ada
418@end ifset
419
420@subheading DIRECTIVE STATUS CODES:
421
422@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
423@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
424@code{@value{RPREFIX}INVALID_ID} - invalid scheduler id@*
425@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
426set of processors owned by the scheduler
427
428@subheading DESCRIPTION:
429
430Returns the processor set owned by the scheduler in @code{cpuset}.  A set bit
431in the processor set means that this processor is owned by the scheduler and a
432cleared bit means the opposite.
433
434@subheading NOTES:
435
436None.
437
438@c
439@c rtems_task_get_scheduler
440@c
441@page
442@subsection TASK_GET_SCHEDULER - Get scheduler of a task
443
444@subheading CALLING SEQUENCE:
445
446@ifset is-C
447@example
448rtems_status_code rtems_task_get_scheduler(
449  rtems_id  task_id,
450  rtems_id *scheduler_id
451);
452@end example
453@end ifset
454
455@ifset is-Ada
456@end ifset
457
458@subheading DIRECTIVE STATUS CODES:
459
460@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
461@code{@value{RPREFIX}INVALID_ADDRESS} - @code{scheduler_id} is NULL@*
462@code{@value{RPREFIX}INVALID_ID} - invalid task id
463
464@subheading DESCRIPTION:
465
466Returns the scheduler identifier of a task identified by @code{task_id} in
467@code{scheduler_id}.
468
469@subheading NOTES:
470
471None.
472
473@c
474@c rtems_task_set_scheduler
475@c
476@page
477@subsection TASK_SET_SCHEDULER - Set scheduler of a task
478
479@subheading CALLING SEQUENCE:
480
481@ifset is-C
482@example
483rtems_status_code rtems_task_set_scheduler(
484  rtems_id task_id,
485  rtems_id scheduler_id
486);
487@end example
488@end ifset
489
490@ifset is-Ada
491@end ifset
492
493@subheading DIRECTIVE STATUS CODES:
494
495@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
496@code{@value{RPREFIX}INVALID_ID} - invalid task or scheduler id@*
497@code{@value{RPREFIX}INCORRECT_STATE} - the task is in the wrong state to
498perform a scheduler change
499
500@subheading DESCRIPTION:
501
502Sets the scheduler of a task identified by @code{task_id} to the scheduler
503identified by @code{scheduler_id}.  The scheduler of a task is initialized to
504the scheduler of the task that created it.
505
506@subheading NOTES:
507
508None.
509
510@subheading EXAMPLE:
511
512@example
513@group
514#include <rtems.h>
515#include <assert.h>
516
517void task(rtems_task_argument arg);
518
519void example(void)
520@{
521  rtems_status_code sc;
522  rtems_id          task_id;
523  rtems_id          scheduler_id;
524  rtems_name        scheduler_name;
525
526  scheduler_name = rtems_build_name('W', 'O', 'R', 'K');
527
528  sc = rtems_scheduler_ident(scheduler_name, &scheduler_id);
529  assert(sc == RTEMS_SUCCESSFUL);
530
531  sc = rtems_task_create(
532    rtems_build_name('T', 'A', 'S', 'K'),
533    1,
534    RTEMS_MINIMUM_STACK_SIZE,
535    RTEMS_DEFAULT_MODES,
536    RTEMS_DEFAULT_ATTRIBUTES,
537    &task_id
538  );
539  assert(sc == RTEMS_SUCCESSFUL);
540
541  sc = rtems_task_set_scheduler(task_id, scheduler_id);
542  assert(sc == RTEMS_SUCCESSFUL);
543
544  sc = rtems_task_start(task_id, task, 0);
545  assert(sc == RTEMS_SUCCESSFUL);
546@}
547@end group
548@end example
549
550@c
551@c rtems_task_get_affinity
552@c
553@page
554@subsection TASK_GET_AFFINITY - Get task processor affinity
555
556@subheading CALLING SEQUENCE:
557
558@ifset is-C
559@example
560rtems_status_code rtems_task_get_affinity(
561  rtems_id   id,
562  size_t     cpusetsize,
563  cpu_set_t *cpuset
564);
565@end example
566@end ifset
567
568@ifset is-Ada
569@end ifset
570
571@subheading DIRECTIVE STATUS CODES:
572
573@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
574@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
575@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
576@code{@value{RPREFIX}INVALID_NUMBER} - the affinity set buffer is too small for
577the current processor affinity set of the task
578
579@subheading DESCRIPTION:
580
581Returns the current processor affinity set of the task in @code{cpuset}.  A set
582bit in the affinity set means that the task can execute on this processor and a
583cleared bit means the opposite.
584
585@subheading NOTES:
586
587None.
588
589@c
590@c rtems_task_set_affinity
591@c
592@page
593@subsection TASK_SET_AFFINITY - Set task processor affinity
594
595@subheading CALLING SEQUENCE:
596
597@ifset is-C
598@example
599rtems_status_code rtems_task_set_affinity(
600  rtems_id         id,
601  size_t           cpusetsize,
602  const cpu_set_t *cpuset
603);
604@end example
605@end ifset
606
607@ifset is-Ada
608@end ifset
609
610@subheading DIRECTIVE STATUS CODES:
611
612@code{@value{RPREFIX}SUCCESSFUL} - successful operation@*
613@code{@value{RPREFIX}INVALID_ADDRESS} - @code{cpuset} is NULL@*
614@code{@value{RPREFIX}INVALID_ID} - invalid task id@*
615@code{@value{RPREFIX}INVALID_NUMBER} - invalid processor affinity set
616
617@subheading DESCRIPTION:
618
619Sets the processor affinity set for the task specified by @code{cpuset}.  A set
620bit in the affinity set means that the task can execute on this processor and a
621cleared bit means the opposite.
622
623@subheading NOTES:
624
625This function will not change the scheduler of the task.  The intersection of
626the processor affinity set and the set of processors owned by the scheduler of
627the task must be non-empty.  It is not an error if the processor affinity set
628contains processors that are not part of the set of processors owned by the
629scheduler instance of the task.  A task will simply not run under normal
630circumstances on these processors since the scheduler ignores them.  Some
631locking protocols may temporarily use processors that are not included in the
632processor affinity set of the task.  It is also not an error if the processor
633affinity set contains processors that are not part of the system.
Note: See TracBrowser for help on using the repository browser.