source: rtems-docs/c_user/multiprocessing.rst @ f916fca

4.115
Last change on this file since f916fca was d389819, checked in by Amar Takhar <amar@…>, on 01/18/16 at 05:37:40

Convert all Unicode to ASCII(128)

  • Property mode set to 100644
File size: 20.6 KB
Line 
1Multiprocessing Manager
2#######################
3
4.. index:: multiprocessing
5
6Introduction
7============
8
9In multiprocessor real-time systems, new
10requirements, such as sharing data and global resources between
11processors, are introduced.  This requires an efficient and
12reliable communications vehicle which allows all processors to
13communicate with each other as necessary.  In addition, the
14ramifications of multiple processors affect each and every
15characteristic of a real-time system, almost always making them
16more complicated.
17
18RTEMS addresses these issues by providing simple and
19flexible real-time multiprocessing capabilities.  The executive
20easily lends itself to both tightly-coupled and loosely-coupled
21configurations of the target system hardware.  In addition,
22RTEMS supports systems composed of both homogeneous and
23heterogeneous mixtures of processors and target boards.
24
25A major design goal of the RTEMS executive was to
26transcend the physical boundaries of the target hardware
27configuration.  This goal is achieved by presenting the
28application software with a logical view of the target system
29where the boundaries between processor nodes are transparent.
30As a result, the application developer may designate objects
31such as tasks, queues, events, signals, semaphores, and memory
32blocks as global objects.  These global objects may then be
33accessed by any task regardless of the physical location of the
34object and the accessing task.  RTEMS automatically determines
35that the object being accessed resides on another processor and
36performs the actions required to access the desired object.
37Simply stated, RTEMS allows the entire system, both hardware and
38software, to be viewed logically as a single system.
39
40Background
41==========
42
43.. index:: multiprocessing topologies
44
45RTEMS makes no assumptions regarding the connection
46media or topology of a multiprocessor system.  The tasks which
47compose a particular application can be spread among as many
48processors as needed to satisfy the application's timing
49requirements.  The application tasks can interact using a subset
50of the RTEMS directives as if they were on the same processor.
51These directives allow application tasks to exchange data,
52communicate, and synchronize regardless of which processor they
53reside upon.
54
55The RTEMS multiprocessor execution model is multiple
56instruction streams with multiple data streams (MIMD).  This
57execution model has each of the processors executing code
58independent of the other processors.  Because of this
59parallelism, the application designer can more easily guarantee
60deterministic behavior.
61
62By supporting heterogeneous environments, RTEMS
63allows the systems designer to select the most efficient
64processor for each subsystem of the application.  Configuring
65RTEMS for a heterogeneous environment is no more difficult than
66for a homogeneous one.  In keeping with RTEMS philosophy of
67providing transparent physical node boundaries, the minimal
68heterogeneous processing required is isolated in the MPCI layer.
69
70Nodes
71-----
72.. index:: nodes, definition
73
74A processor in a RTEMS system is referred to as a
75node.  Each node is assigned a unique non-zero node number by
76the application designer.  RTEMS assumes that node numbers are
77assigned consecutively from one to the ``maximum_nodes``
78configuration parameter.  The node
79number, node, and the maximum number of nodes, maximum_nodes, in
80a system are found in the Multiprocessor Configuration Table.
81The maximum_nodes field and the number of global objects,
82maximum_global_objects, is required to be the same on all nodes
83in a system.
84
85The node number is used by RTEMS to identify each
86node when performing remote operations.  Thus, the
87Multiprocessor Communications Interface Layer (MPCI) must be
88able to route messages based on the node number.
89
90Global Objects
91--------------
92.. index:: global objects, definition
93
94All RTEMS objects which are created with the GLOBAL
95attribute will be known on all other nodes.  Global objects can
96be referenced from any node in the system, although certain
97directive specific restrictions (e.g. one cannot delete a remote
98object) may apply.  A task does not have to be global to perform
99operations involving remote objects.  The maximum number of
100global objects is the system is user configurable and can be
101found in the maximum_global_objects field in the Multiprocessor
102Configuration Table.  The distribution of tasks to processors is
103performed during the application design phase.  Dynamic task
104relocation is not supported by RTEMS.
105
106Global Object Table
107-------------------
108.. index:: global objects table
109
110RTEMS maintains two tables containing object
111information on every node in a multiprocessor system: a local
112object table and a global object table.  The local object table
113on each node is unique and contains information for all objects
114created on this node whether those objects are local or global.
115The global object table contains information regarding all
116global objects in the system and, consequently, is the same on
117every node.
118
119Since each node must maintain an identical copy of
120the global object table,  the maximum number of entries in each
121copy of the table must be the same.  The maximum number of
122entries in each copy is determined by the
123maximum_global_objects parameter in the Multiprocessor
124Configuration Table.  This parameter, as well as the
125maximum_nodes parameter, is required to be the same on all
126nodes.  To maintain consistency among the table copies, every
127node in the system must be informed of the creation or deletion
128of a global object.
129
130Remote Operations
131-----------------
132.. index:: MPCI and remote operations
133
134When an application performs an operation on a remote
135global object, RTEMS must generate a Remote Request (RQ) message
136and send it to the appropriate node.  After completing the
137requested operation, the remote node will build a Remote
138Response (RR) message and send it to the originating node.
139Messages generated as a side-effect of a directive (such as
140deleting a global task) are known as Remote Processes (RP) and
141do not require the receiving node to respond.
142
143Other than taking slightly longer to execute
144directives on remote objects, the application is unaware of the
145location of the objects it acts upon.  The exact amount of
146overhead required for a remote operation is dependent on the
147media connecting the nodes and, to a lesser degree, on the
148efficiency of the user-provided MPCI routines.
149
150The following shows the typical transaction sequence
151during a remote application:
152
153# The application issues a directive accessing a
154  remote global object.
155
156# RTEMS determines the node on which the object
157  resides.
158
159# RTEMS calls the user-provided MPCI routine
160  GET_PACKET to obtain a packet in which to build a RQ message.
161
162# After building a message packet, RTEMS calls the
163  user-provided MPCI routine SEND_PACKET to transmit the packet to
164  the node on which the object resides (referred to as the
165  destination node).
166
167# The calling task is blocked until the RR message
168  arrives, and control of the processor is transferred to another
169  task.
170
171# The MPCI layer on the destination node senses the
172  arrival of a packet (commonly in an ISR), and calls the``rtems_multiprocessing_announce``
173  directive.  This directive readies the Multiprocessing Server.
174
175# The Multiprocessing Server calls the user-provided
176  MPCI routine RECEIVE_PACKET, performs the requested operation,
177  builds an RR message, and returns it to the originating node.
178
179# The MPCI layer on the originating node senses the
180  arrival of a packet (typically via an interrupt), and calls the RTEMS``rtems_multiprocessing_announce`` directive.  This directive
181  readies the Multiprocessing Server.
182
183# The Multiprocessing Server calls the user-provided
184  MPCI routine RECEIVE_PACKET, readies the original requesting
185  task, and blocks until another packet arrives.  Control is
186  transferred to the original task which then completes processing
187  of the directive.
188
189If an uncorrectable error occurs in the user-provided
190MPCI layer, the fatal error handler should be invoked.  RTEMS
191assumes the reliable transmission and reception of messages by
192the MPCI and makes no attempt to detect or correct errors.
193
194Proxies
195-------
196.. index:: proxy, definition
197
198A proxy is an RTEMS data structure which resides on a
199remote node and is used to represent a task which must block as
200part of a remote operation. This action can occur as part of the``rtems_semaphore_obtain`` and``rtems_message_queue_receive`` directives.  If the
201object were local, the task's control block would be available
202for modification to indicate it was blocking on a message queue
203or semaphore.  However, the task's control block resides only on
204the same node as the task.  As a result, the remote node must
205allocate a proxy to represent the task until it can be readied.
206
207The maximum number of proxies is defined in the
208Multiprocessor Configuration Table.  Each node in a
209multiprocessor system may require a different number of proxies
210to be configured.  The distribution of proxy control blocks is
211application dependent and is different from the distribution of
212tasks.
213
214Multiprocessor Configuration Table
215----------------------------------
216
217The Multiprocessor Configuration Table contains
218information needed by RTEMS when used in a multiprocessor
219system.  This table is discussed in detail in the section
220Multiprocessor Configuration Table of the Configuring a System
221chapter.
222
223Multiprocessor Communications Interface Layer
224=============================================
225
226The Multiprocessor Communications Interface Layer
227(MPCI) is a set of user-provided procedures which enable the
228nodes in a multiprocessor system to communicate with one
229another.  These routines are invoked by RTEMS at various times
230in the preparation and processing of remote requests.
231Interrupts are enabled when an MPCI procedure is invoked.  It is
232assumed that if the execution mode and/or interrupt level are
233altered by the MPCI layer, that they will be restored prior to
234returning to RTEMS... index:: MPCI, definition
235
236The MPCI layer is responsible for managing a pool of
237buffers called packets and for sending these packets between
238system nodes.  Packet buffers contain the messages sent between
239the nodes.  Typically, the MPCI layer will encapsulate the
240packet within an envelope which contains the information needed
241by the MPCI layer.  The number of packets available is dependent
242on the MPCI layer implementation... index:: MPCI entry points
243
244The entry points to the routines in the user's MPCI
245layer should be placed in the Multiprocessor Communications
246Interface Table.  The user must provide entry points for each of
247the following table entries in a multiprocessor system:
248
249- initialization        initialize the MPCI
250
251- get_packet    obtain a packet buffer
252
253- return_packet         return a packet buffer
254
255- send_packet   send a packet to another node
256
257- receive_packet        called to get an arrived packet
258
259A packet is sent by RTEMS in each of the following situations:
260
261- an RQ is generated on an originating node;
262
263- an RR is generated on a destination node;
264
265- a global object is created;
266
267- a global object is deleted;
268
269- a local task blocked on a remote object is deleted;
270
271- during system initialization to check for system consistency.
272
273If the target hardware supports it, the arrival of a
274packet at a node may generate an interrupt.  Otherwise, the
275real-time clock ISR can check for the arrival of a packet.  In
276any case, the``rtems_multiprocessing_announce`` directive must be called
277to announce the arrival of a packet.  After exiting the ISR,
278control will be passed to the Multiprocessing Server to process
279the packet.  The Multiprocessing Server will call the get_packet
280entry to obtain a packet buffer and the receive_entry entry to
281copy the message into the buffer obtained.
282
283INITIALIZATION
284--------------
285
286The INITIALIZATION component of the user-provided
287MPCI layer is called as part of the ``rtems_initialize_executive``
288directive to initialize the MPCI layer and associated hardware.
289It is invoked immediately after all of the device drivers have
290been initialized.  This component should be adhere to the
291following prototype:.. index:: rtems_mpci_entry
292
293.. code:: c
294
295    rtems_mpci_entry user_mpci_initialization(
296    rtems_configuration_table \*configuration
297    );
298
299where configuration is the address of the user's
300Configuration Table.  Operations on global objects cannot be
301performed until this component is invoked.  The INITIALIZATION
302component is invoked only once in the life of any system.  If
303the MPCI layer cannot be successfully initialized, the fatal
304error manager should be invoked by this routine.
305
306One of the primary functions of the MPCI layer is to
307provide the executive with packet buffers.  The INITIALIZATION
308routine must create and initialize a pool of packet buffers.
309There must be enough packet buffers so RTEMS can obtain one
310whenever needed.
311
312GET_PACKET
313----------
314
315The GET_PACKET component of the user-provided MPCI
316layer is called when RTEMS must obtain a packet buffer to send
317or broadcast a message.  This component should be adhere to the
318following prototype:
319.. code:: c
320
321    rtems_mpci_entry user_mpci_get_packet(
322    rtems_packet_prefix \**packet
323    );
324
325where packet is the address of a pointer to a packet.
326This routine always succeeds and, upon return, packet will
327contain the address of a packet.  If for any reason, a packet
328cannot be successfully obtained, then the fatal error manager
329should be invoked.
330
331RTEMS has been optimized to avoid the need for
332obtaining a packet each time a message is sent or broadcast.
333For example, RTEMS sends response messages (RR) back to the
334originator in the same packet in which the request message (RQ)
335arrived.
336
337RETURN_PACKET
338-------------
339
340The RETURN_PACKET component of the user-provided MPCI
341layer is called when RTEMS needs to release a packet to the free
342packet buffer pool.  This component should be adhere to the
343following prototype:
344.. code:: c
345
346    rtems_mpci_entry user_mpci_return_packet(
347    rtems_packet_prefix \*packet
348    );
349
350where packet is the address of a packet.  If the
351packet cannot be successfully returned, the fatal error manager
352should be invoked.
353
354RECEIVE_PACKET
355--------------
356
357The RECEIVE_PACKET component of the user-provided
358MPCI layer is called when RTEMS needs to obtain a packet which
359has previously arrived.  This component should be adhere to the
360following prototype:
361.. code:: c
362
363    rtems_mpci_entry user_mpci_receive_packet(
364    rtems_packet_prefix \**packet
365    );
366
367where packet is a pointer to the address of a packet
368to place the message from another node.  If a message is
369available, then packet will contain the address of the message
370from another node.  If no messages are available, this entry
371packet should contain NULL.
372
373SEND_PACKET
374-----------
375
376The SEND_PACKET component of the user-provided MPCI
377layer is called when RTEMS needs to send a packet containing a
378message to another node.  This component should be adhere to the
379following prototype:
380.. code:: c
381
382    rtems_mpci_entry user_mpci_send_packet(
383    uint32_t               node,
384    rtems_packet_prefix  \**packet
385    );
386
387where node is the node number of the destination and packet is the
388address of a packet which containing a message.  If the packet cannot
389be successfully sent, the fatal error manager should be invoked.
390
391If node is set to zero, the packet is to be
392broadcasted to all other nodes in the system.  Although some
393MPCI layers will be built upon hardware which support a
394broadcast mechanism, others may be required to generate a copy
395of the packet for each node in the system.
396
397.. COMMENT: XXX packet_prefix structure needs to be defined in this document
398
399Many MPCI layers use the ``packet_length`` field of the``rtems_packet_prefix`` portion
400of the packet to avoid sending unnecessary data.  This is especially
401useful if the media connecting the nodes is relatively slow.
402
403The ``to_convert`` field of the ``rtems_packet_prefix`` portion of the
404packet indicates how much of the packet in 32-bit units may require conversion
405in a heterogeneous system.
406
407Supporting Heterogeneous Environments
408-------------------------------------
409.. index:: heterogeneous multiprocessing
410
411Developing an MPCI layer for a heterogeneous system
412requires a thorough understanding of the differences between the
413processors which comprise the system.  One difficult problem is
414the varying data representation schemes used by different
415processor types.  The most pervasive data representation problem
416is the order of the bytes which compose a data entity.
417Processors which place the least significant byte at the
418smallest address are classified as little endian processors.
419Little endian byte-ordering is shown below:
420
421.. code:: c
422
423    +---------------+----------------+---------------+----------------+
424    |               |                |               |                |
425    |    Byte 3     |     Byte 2     |    Byte 1     |    Byte 0      |
426    |               |                |               |                |
427    +---------------+----------------+---------------+----------------+
428
429Conversely, processors which place the most
430significant byte at the smallest address are classified as big
431endian processors.  Big endian byte-ordering is shown below:
432.. code:: c
433
434    +---------------+----------------+---------------+----------------+
435    |               |                |               |                |
436    |    Byte 0     |     Byte 1     |    Byte 2     |    Byte 3      |
437    |               |                |               |                |
438    +---------------+----------------+---------------+----------------+
439
440Unfortunately, sharing a data structure between big
441endian and little endian processors requires translation into a
442common endian format.  An application designer typically chooses
443the common endian format to minimize conversion overhead.
444
445Another issue in the design of shared data structures
446is the alignment of data structure elements.  Alignment is both
447processor and compiler implementation dependent.  For example,
448some processors allow data elements to begin on any address
449boundary, while others impose restrictions.  Common restrictions
450are that data elements must begin on either an even address or
451on a long word boundary.  Violation of these restrictions may
452cause an exception or impose a performance penalty.
453
454Other issues which commonly impact the design of
455shared data structures include the representation of floating
456point numbers, bit fields, decimal data, and character strings.
457In addition, the representation method for negative integers
458could be one's or two's complement.  These factors combine to
459increase the complexity of designing and manipulating data
460structures shared between processors.
461
462RTEMS addressed these issues in the design of the
463packets used to communicate between nodes.  The RTEMS packet
464format is designed to allow the MPCI layer to perform all
465necessary conversion without burdening the developer with the
466details of the RTEMS packet format.  As a result, the MPCI layer
467must be aware of the following:
468
469- All packets must begin on a four byte boundary.
470
471- Packets are composed of both RTEMS and application data.  All RTEMS data
472  is treated as 32-bit unsigned quantities and is in the first ``to_convert``
473  32-bit quantities of the packet.  The ``to_convert`` field is part of the``rtems_packet_prefix`` portion of the packet.
474
475- The RTEMS data component of the packet must be in native
476  endian format.  Endian conversion may be performed by either the
477  sending or receiving MPCI layer.
478
479- RTEMS makes no assumptions regarding the application
480  data component of the packet.
481
482Operations
483==========
484
485Announcing a Packet
486-------------------
487
488The ``rtems_multiprocessing_announce`` directive is called by
489the MPCI layer to inform RTEMS that a packet has arrived from
490another node.  This directive can be called from an interrupt
491service routine or from within a polling routine.
492
493Directives
494==========
495
496This section details the additional directives
497required to support RTEMS in a multiprocessor configuration.  A
498subsection is dedicated to each of this manager's directives and
499describes the calling sequence, related constants, usage, and
500status codes.
501
502MULTIPROCESSING_ANNOUNCE - Announce the arrival of a packet
503-----------------------------------------------------------
504.. index:: announce arrival of package
505
506**CALLING SEQUENCE:**
507
508.. index:: rtems_multiprocessing_announce
509
510.. code:: c
511
512    void rtems_multiprocessing_announce( void );
513
514**DIRECTIVE STATUS CODES:**
515
516NONE
517
518**DESCRIPTION:**
519
520This directive informs RTEMS that a multiprocessing
521communications packet has arrived from another node.  This
522directive is called by the user-provided MPCI, and is only used
523in multiprocessor configurations.
524
525**NOTES:**
526
527This directive is typically called from an ISR.
528
529This directive will almost certainly cause the
530calling task to be preempted.
531
532This directive does not generate activity on remote nodes.
533
534.. COMMENT: COPYRIGHT (c) 2014.
535
536.. COMMENT: On-Line Applications Research Corporation (OAR).
537
538.. COMMENT: All rights reserved.
539
Note: See TracBrowser for help on using the repository browser.