source: rtems/doc/user/mp.t @ 1b03eed

4.8
Last change on this file since 1b03eed was 1b03eed, checked in by Glenn Humphrey <glenn.humphrey@…>, on 10/26/07 at 21:34:57

2007-10-26 Glenn Humphrey <glenn.humphrey@…>

  • user/rtmon.t: Fix report output.

2007-10-25 Glenn Humphrey <glenn.humphrey@…>

  • user/barrier.t, user/clock.t, user/concepts.t, user/cpuuse.t, user/init.t, user/intr.t, user/io.t, user/mp.t, user/rtmon.t, user/sem.t, user/stackchk.t, user/task.t, user/timer.t: Updated the Ada documentation to reflect the current binding.
  • Property mode set to 100644
File size: 21.4 KB
Line 
1@c
2@c  COPYRIGHT (c) 1988-2007.
3@c  On-Line Applications Research Corporation (OAR).
4@c  All rights reserved.
5@c
6@c  $Id$
7@c
8
9@chapter Multiprocessing Manager
10
11@cindex multiprocessing
12
13@section Introduction
14
15In multiprocessor real-time systems, new
16requirements, such as sharing data and global resources between
17processors, are introduced.  This requires an efficient and
18reliable communications vehicle which allows all processors to
19communicate with each other as necessary.  In addition, the
20ramifications of multiple processors affect each and every
21characteristic of a real-time system, almost always making them
22more complicated.
23
24RTEMS addresses these issues by providing simple and
25flexible real-time multiprocessing capabilities.  The executive
26easily lends itself to both tightly-coupled and loosely-coupled
27configurations of the target system hardware.  In addition,
28RTEMS supports systems composed of both homogeneous and
29heterogeneous mixtures of processors and target boards.
30
31A major design goal of the RTEMS executive was to
32transcend the physical boundaries of the target hardware
33configuration.  This goal is achieved by presenting the
34application software with a logical view of the target system
35where the boundaries between processor nodes are transparent.
36As a result, the application developer may designate objects
37such as tasks, queues, events, signals, semaphores, and memory
38blocks as global objects.  These global objects may then be
39accessed by any task regardless of the physical location of the
40object and the accessing task.  RTEMS automatically determines
41that the object being accessed resides on another processor and
42performs the actions required to access the desired object.
43Simply stated, RTEMS allows the entire system, both hardware and
44software, to be viewed logically as a single system.
45
46@section Background
47
48@cindex multiprocessing topologies
49
50RTEMS makes no assumptions regarding the connection
51media or topology of a multiprocessor system.  The tasks which
52compose a particular application can be spread among as many
53processors as needed to satisfy the application's timing
54requirements.  The application tasks can interact using a subset
55of the RTEMS directives as if they were on the same processor.
56These directives allow application tasks to exchange data,
57communicate, and synchronize regardless of which processor they
58reside upon.
59
60The RTEMS multiprocessor execution model is multiple
61instruction streams with multiple data streams (MIMD).  This
62execution model has each of the processors executing code
63independent of the other processors.  Because of this
64parallelism, the application designer can more easily guarantee
65deterministic behavior.
66
67By supporting heterogeneous environments, RTEMS
68allows the systems designer to select the most efficient
69processor for each subsystem of the application.  Configuring
70RTEMS for a heterogeneous environment is no more difficult than
71for a homogeneous one.  In keeping with RTEMS philosophy of
72providing transparent physical node boundaries, the minimal
73heterogeneous processing required is isolated in the MPCI layer.
74
75@subsection Nodes
76
77@cindex nodes, definition
78
79A processor in a RTEMS system is referred to as a
80node.  Each node is assigned a unique non-zero node number by
81the application designer.  RTEMS assumes that node numbers are
82assigned consecutively from one to the @code{maximum_nodes}
83configuration parameter.  The node
84number, node, and the maximum number of nodes, maximum_nodes, in
85a system are found in the Multiprocessor Configuration Table.
86The maximum_nodes field and the number of global objects,
87maximum_global_objects, is required to be the same on all nodes
88in a system.
89
90The node number is used by RTEMS to identify each
91node when performing remote operations.  Thus, the
92Multiprocessor Communications Interface Layer (MPCI) must be
93able to route messages based on the node number.
94
95@subsection Global Objects
96
97@cindex global objects, definition
98
99All RTEMS objects which are created with the GLOBAL
100attribute will be known on all other nodes.  Global objects can
101be referenced from any node in the system, although certain
102directive specific restrictions (e.g. one cannot delete a remote
103object) may apply.  A task does not have to be global to perform
104operations involving remote objects.  The maximum number of
105global objects is the system is user configurable and can be
106found in the maximum_global_objects field in the Multiprocessor
107Configuration Table.  The distribution of tasks to processors is
108performed during the application design phase.  Dynamic task
109relocation is not supported by RTEMS.
110
111@subsection Global Object Table
112
113@cindex global objects table
114
115RTEMS maintains two tables containing object
116information on every node in a multiprocessor system: a local
117object table and a global object table.  The local object table
118on each node is unique and contains information for all objects
119created on this node whether those objects are local or global.
120The global object table contains information regarding all
121global objects in the system and, consequently, is the same on
122every node.
123
124Since each node must maintain an identical copy of
125the global object table,  the maximum number of entries in each
126copy of the table must be the same.  The maximum number of
127entries in each copy is determined by the
128maximum_global_objects parameter in the Multiprocessor
129Configuration Table.  This parameter, as well as the
130maximum_nodes parameter, is required to be the same on all
131nodes.  To maintain consistency among the table copies, every
132node in the system must be informed of the creation or deletion
133of a global object.
134
135@subsection Remote Operations
136
137@cindex MPCI and remote operations
138
139When an application performs an operation on a remote
140global object, RTEMS must generate a Remote Request (RQ) message
141and send it to the appropriate node.  After completing the
142requested operation, the remote node will build a Remote
143Response (RR) message and send it to the originating node.
144Messages generated as a side-effect of a directive (such as
145deleting a global task) are known as Remote Processes (RP) and
146do not require the receiving node to respond.
147
148Other than taking slightly longer to execute
149directives on remote objects, the application is unaware of the
150location of the objects it acts upon.  The exact amount of
151overhead required for a remote operation is dependent on the
152media connecting the nodes and, to a lesser degree, on the
153efficiency of the user-provided MPCI routines.
154
155The following shows the typical transaction sequence
156during a remote application:
157
158@enumerate
159
160@item The application issues a directive accessing a
161remote global object.
162
163@item RTEMS determines the node on which the object
164resides.
165
166@item RTEMS calls the user-provided MPCI routine
167GET_PACKET to obtain a packet in which to build a RQ message.
168
169@item After building a message packet, RTEMS calls the
170user-provided MPCI routine SEND_PACKET to transmit the packet to
171the node on which the object resides (referred to as the
172destination node).
173
174@item The calling task is blocked until the RR message
175arrives, and control of the processor is transferred to another
176task.
177
178@item The MPCI layer on the destination node senses the
179arrival of a packet (commonly in an ISR), and calls the
180@code{@value{DIRPREFIX}multiprocessing_announce}
181directive.  This directive readies the Multiprocessing Server.
182
183@item The Multiprocessing Server calls the user-provided
184MPCI routine RECEIVE_PACKET, performs the requested operation,
185builds an RR message, and returns it to the originating node.
186
187@item The MPCI layer on the originating node senses the
188arrival of a packet (typically via an interrupt), and calls the RTEMS
189@code{@value{DIRPREFIX}multiprocessing_announce} directive.  This directive
190readies the Multiprocessing Server.
191
192@item The Multiprocessing Server calls the user-provided
193MPCI routine RECEIVE_PACKET, readies the original requesting
194task, and blocks until another packet arrives.  Control is
195transferred to the original task which then completes processing
196of the directive.
197
198@end enumerate
199
200If an uncorrectable error occurs in the user-provided
201MPCI layer, the fatal error handler should be invoked.  RTEMS
202assumes the reliable transmission and reception of messages by
203the MPCI and makes no attempt to detect or correct errors.
204
205@subsection Proxies
206
207@cindex proxy, definition
208
209A proxy is an RTEMS data structure which resides on a
210remote node and is used to represent a task which must block as
211part of a remote operation. This action can occur as part of the
212@code{@value{DIRPREFIX}semaphore_obtain} and
213@code{@value{DIRPREFIX}message_queue_receive} directives.  If the
214object were local, the task's control block would be available
215for modification to indicate it was blocking on a message queue
216or semaphore.  However, the task's control block resides only on
217the same node as the task.  As a result, the remote node must
218allocate a proxy to represent the task until it can be readied.
219
220The maximum number of proxies is defined in the
221Multiprocessor Configuration Table.  Each node in a
222multiprocessor system may require a different number of proxies
223to be configured.  The distribution of proxy control blocks is
224application dependent and is different from the distribution of
225tasks.
226
227@subsection Multiprocessor Configuration Table
228
229The Multiprocessor Configuration Table contains
230information needed by RTEMS when used in a multiprocessor
231system.  This table is discussed in detail in the section
232Multiprocessor Configuration Table of the Configuring a System
233chapter.
234
235@section Multiprocessor Communications Interface Layer
236
237The Multiprocessor Communications Interface Layer
238(MPCI) is a set of user-provided procedures which enable the
239nodes in a multiprocessor system to communicate with one
240another.  These routines are invoked by RTEMS at various times
241in the preparation and processing of remote requests.
242Interrupts are enabled when an MPCI procedure is invoked.  It is
243assumed that if the execution mode and/or interrupt level are
244altered by the MPCI layer, that they will be restored prior to
245returning to RTEMS.
246
247@cindex MPCI, definition
248
249The MPCI layer is responsible for managing a pool of
250buffers called packets and for sending these packets between
251system nodes.  Packet buffers contain the messages sent between
252the nodes.  Typically, the MPCI layer will encapsulate the
253packet within an envelope which contains the information needed
254by the MPCI layer.  The number of packets available is dependent
255on the MPCI layer implementation.
256
257@cindex MPCI entry points
258
259The entry points to the routines in the user's MPCI
260layer should be placed in the Multiprocessor Communications
261Interface Table.  The user must provide entry points for each of
262the following table entries in a multiprocessor system:
263
264@itemize @bullet
265@item initialization    initialize the MPCI
266@item get_packet        obtain a packet buffer
267@item return_packet     return a packet buffer
268@item send_packet       send a packet to another node
269@item receive_packet    called to get an arrived packet
270@end itemize
271
272A packet is sent by RTEMS in each of the following situations:
273
274@itemize @bullet
275@item an RQ is generated on an originating node;
276@item an RR is generated on a destination node;
277@item a global object is created;
278@item a global object is deleted;
279@item a local task blocked on a remote object is deleted;
280@item during system initialization to check for system consistency.
281@end itemize
282
283If the target hardware supports it, the arrival of a
284packet at a node may generate an interrupt.  Otherwise, the
285real-time clock ISR can check for the arrival of a packet.  In
286any case, the
287@code{@value{DIRPREFIX}multiprocessing_announce} directive must be called
288to announce the arrival of a packet.  After exiting the ISR,
289control will be passed to the Multiprocessing Server to process
290the packet.  The Multiprocessing Server will call the get_packet
291entry to obtain a packet buffer and the receive_entry entry to
292copy the message into the buffer obtained.
293
294@subsection INITIALIZATION
295
296The INITIALIZATION component of the user-provided
297MPCI layer is called as part of the @code{@value{DIRPREFIX}initialize_executive}
298directive to initialize the MPCI layer and associated hardware.
299It is invoked immediately after all of the device drivers have
300been initialized.  This component should be adhere to the
301following prototype:
302
303@ifset is-C
304@findex rtems_mpci_entry
305@example
306@group
307rtems_mpci_entry user_mpci_initialization(
308  rtems_configuration_table *configuration
309);
310@end group
311@end example
312@end ifset
313
314@ifset is-Ada
315@example
316NOT SUPPORTED FROM Ada BINDING
317@end example
318@end ifset
319
320where configuration is the address of the user's
321Configuration Table.  Operations on global objects cannot be
322performed until this component is invoked.  The INITIALIZATION
323component is invoked only once in the life of any system.  If
324the MPCI layer cannot be successfully initialized, the fatal
325error manager should be invoked by this routine.
326
327One of the primary functions of the MPCI layer is to
328provide the executive with packet buffers.  The INITIALIZATION
329routine must create and initialize a pool of packet buffers.
330There must be enough packet buffers so RTEMS can obtain one
331whenever needed.
332
333@subsection GET_PACKET
334
335The GET_PACKET component of the user-provided MPCI
336layer is called when RTEMS must obtain a packet buffer to send
337or broadcast a message.  This component should be adhere to the
338following prototype:
339
340@ifset is-C
341@example
342@group
343rtems_mpci_entry user_mpci_get_packet(
344  rtems_packet_prefix **packet
345);
346@end group
347@end example
348@end ifset
349
350@ifset is-Ada
351@example
352NOT SUPPORTED FROM Ada BINDING
353@end example
354@end ifset
355
356where packet is the address of a pointer to a packet.
357This routine always succeeds and, upon return, packet will
358contain the address of a packet.  If for any reason, a packet
359cannot be successfully obtained, then the fatal error manager
360should be invoked.
361
362RTEMS has been optimized to avoid the need for
363obtaining a packet each time a message is sent or broadcast.
364For example, RTEMS sends response messages (RR) back to the
365originator in the same packet in which the request message (RQ)
366arrived.
367
368@subsection RETURN_PACKET
369
370The RETURN_PACKET component of the user-provided MPCI
371layer is called when RTEMS needs to release a packet to the free
372packet buffer pool.  This component should be adhere to the
373following prototype:
374
375@ifset is-C
376@example
377@group
378rtems_mpci_entry user_mpci_return_packet(
379  rtems_packet_prefix *packet
380);
381@end group
382@end example
383@end ifset
384
385@ifset is-Ada
386@example
387NOT SUPPORTED FROM Ada BINDING
388@end example
389@end ifset
390
391where packet is the address of a packet.  If the
392packet cannot be successfully returned, the fatal error manager
393should be invoked.
394
395@subsection RECEIVE_PACKET
396
397The RECEIVE_PACKET component of the user-provided
398MPCI layer is called when RTEMS needs to obtain a packet which
399has previously arrived.  This component should be adhere to the
400following prototype:
401
402@ifset is-C
403@example
404@group
405rtems_mpci_entry user_mpci_receive_packet(
406  rtems_packet_prefix **packet
407);
408@end group
409@end example
410@end ifset
411
412@ifset is-Ada
413@example
414NOT SUPPORTED FROM Ada BINDING
415@end example
416@end ifset
417
418where packet is a pointer to the address of a packet
419to place the message from another node.  If a message is
420available, then packet will contain the address of the message
421from another node.  If no messages are available, this entry
422packet should contain NULL.
423
424@subsection SEND_PACKET
425
426The SEND_PACKET component of the user-provided MPCI
427layer is called when RTEMS needs to send a packet containing a
428message to another node.  This component should be adhere to the
429following prototype:
430
431@ifset is-C
432@example
433@group
434rtems_mpci_entry user_mpci_send_packet(
435  rtems_unsigned32       node,
436  rtems_packet_prefix  **packet
437);
438@end group
439@end example
440@end ifset
441
442@ifset is-Ada
443@example
444NOT SUPPORTED FROM Ada BINDING
445@end example
446@end ifset
447
448where node is the node number of the destination and packet is the
449address of a packet which containing a message.  If the packet cannot
450be successfully sent, the fatal error manager should be invoked.
451
452If node is set to zero, the packet is to be
453broadcasted to all other nodes in the system.  Although some
454MPCI layers will be built upon hardware which support a
455broadcast mechanism, others may be required to generate a copy
456of the packet for each node in the system.
457
458@c XXX packet_prefix structure needs to be defined in this document
459Many MPCI layers use the @code{packet_length} field of the
460@code{@value{DIRPREFIX}packet_prefix} portion
461of the packet to avoid sending unnecessary data.  This is especially
462useful if the media connecting the nodes is relatively slow.
463
464The to_convert field of the MP_packet_prefix portion of the packet indicates
465how much of the packet (in @code{@value{DIRPREFIX}unsigned32}'s) may require
466conversion in a heterogeneous system.
467
468@subsection Supporting Heterogeneous Environments
469
470@cindex heterogeneous multiprocessing
471
472Developing an MPCI layer for a heterogeneous system
473requires a thorough understanding of the differences between the
474processors which comprise the system.  One difficult problem is
475the varying data representation schemes used by different
476processor types.  The most pervasive data representation problem
477is the order of the bytes which compose a data entity.
478Processors which place the least significant byte at the
479smallest address are classified as little endian processors.
480Little endian byte-ordering is shown below:
481
482
483@example
484@group
485+---------------+----------------+---------------+----------------+
486|               |                |               |                |
487|    Byte 3     |     Byte 2     |    Byte 1     |    Byte 0      |
488|               |                |               |                |
489+---------------+----------------+---------------+----------------+
490@end group
491@end example
492
493Conversely, processors which place the most
494significant byte at the smallest address are classified as big
495endian processors.  Big endian byte-ordering is shown below:
496
497@example
498@group
499+---------------+----------------+---------------+----------------+
500|               |                |               |                |
501|    Byte 0     |     Byte 1     |    Byte 2     |    Byte 3      |
502|               |                |               |                |
503+---------------+----------------+---------------+----------------+
504@end group
505@end example
506
507Unfortunately, sharing a data structure between big
508endian and little endian processors requires translation into a
509common endian format.  An application designer typically chooses
510the common endian format to minimize conversion overhead.
511
512Another issue in the design of shared data structures
513is the alignment of data structure elements.  Alignment is both
514processor and compiler implementation dependent.  For example,
515some processors allow data elements to begin on any address
516boundary, while others impose restrictions.  Common restrictions
517are that data elements must begin on either an even address or
518on a long word boundary.  Violation of these restrictions may
519cause an exception or impose a performance penalty.
520
521Other issues which commonly impact the design of
522shared data structures include the representation of floating
523point numbers, bit fields, decimal data, and character strings.
524In addition, the representation method for negative integers
525could be one's or two's complement.  These factors combine to
526increase the complexity of designing and manipulating data
527structures shared between processors.
528
529RTEMS addressed these issues in the design of the
530packets used to communicate between nodes.  The RTEMS packet
531format is designed to allow the MPCI layer to perform all
532necessary conversion without burdening the developer with the
533details of the RTEMS packet format.  As a result, the MPCI layer
534must be aware of the following:
535
536@itemize @bullet
537@item All packets must begin on a four byte boundary.
538
539@item Packets are composed of both RTEMS and application data.
540All RTEMS data is treated as thirty-two (32) bit unsigned
541quantities and is in the first @code{@value{RPREFIX}MINIMUM_UNSIGNED32S_TO_CONVERT}
542thirty-two (32) quantities of the packet.
543
544@item The RTEMS data component of the packet must be in native
545endian format.  Endian conversion may be performed by either the
546sending or receiving MPCI layer.
547
548@item RTEMS makes no assumptions regarding the application
549data component of the packet.
550@end itemize
551
552@section Operations
553
554@subsection Announcing a Packet
555
556The @code{@value{DIRPREFIX}multiprocessing_announce} directive is called by
557the MPCI layer to inform RTEMS that a packet has arrived from
558another node.  This directive can be called from an interrupt
559service routine or from within a polling routine.
560
561@section Directives
562
563This section details the additional directives
564required to support RTEMS in a multiprocessor configuration.  A
565subsection is dedicated to each of this manager's directives and
566describes the calling sequence, related constants, usage, and
567status codes.
568
569@c
570@c
571@c
572@page
573@subsection MULTIPROCESSING_ANNOUNCE - Announce the arrival of a packet
574
575@cindex announce arrival of package
576
577@subheading CALLING SEQUENCE:
578
579@ifset is-C
580@findex rtems_multiprocessing_announce
581@example
582void rtems_multiprocessing_announce( void );
583@end example
584@end ifset
585
586@ifset is-Ada
587@example
588NOT SUPPORTED FROM Ada BINDING
589@end example
590@end ifset
591
592@subheading DIRECTIVE STATUS CODES:
593
594NONE
595
596@subheading DESCRIPTION:
597
598This directive informs RTEMS that a multiprocessing
599communications packet has arrived from another node.  This
600directive is called by the user-provided MPCI, and is only used
601in multiprocessor configurations.
602
603@subheading NOTES:
604
605This directive is typically called from an ISR.
606
607This directive will almost certainly cause the
608calling task to be preempted.
609
610This directive does not generate activity on remote nodes.
Note: See TracBrowser for help on using the repository browser.