source: rtems-docs/c-user/multiprocessing.rst @ 4886d60

5
Last change on this file since 4886d60 was 4886d60, checked in by Sebastian Huber <sebastian.huber@…>, on 01/09/19 at 15:14:04

Use standard format for copyright lines

  • Property mode set to 100644
File size: 21.1 KB
Line 
1.. comment SPDX-License-Identifier: CC-BY-SA-4.0
2
3.. Copyright (C) 1988, 2008 On-Line Applications Research Corporation (OAR)
4.. COMMENT: All rights reserved.
5
6.. index:: multiprocessing
7
8Multiprocessing Manager
9***********************
10
11Introduction
12============
13
14In multiprocessor real-time systems, new requirements, such as sharing data and
15global resources between processors, are introduced.  This requires an
16efficient and reliable communications vehicle which allows all processors to
17communicate with each other as necessary.  In addition, the ramifications of
18multiple processors affect each and every characteristic of a real-time system,
19almost always making them more complicated.
20
21RTEMS addresses these issues by providing simple and flexible real-time
22multiprocessing capabilities.  The executive easily lends itself to both
23tightly-coupled and loosely-coupled configurations of the target system
24hardware.  In addition, RTEMS supports systems composed of both homogeneous and
25heterogeneous mixtures of processors and target boards.
26
27A major design goal of the RTEMS executive was to transcend the physical
28boundaries of the target hardware configuration.  This goal is achieved by
29presenting the application software with a logical view of the target system
30where the boundaries between processor nodes are transparent.  As a result, the
31application developer may designate objects such as tasks, queues, events,
32signals, semaphores, and memory blocks as global objects.  These global objects
33may then be accessed by any task regardless of the physical location of the
34object and the accessing task.  RTEMS automatically determines that the object
35being accessed resides on another processor and performs the actions required
36to access the desired object.  Simply stated, RTEMS allows the entire system,
37both hardware and software, to be viewed logically as a single system.
38
39The directives provided by the  Manager are:
40
41- rtems_multiprocessing_announce_ - A multiprocessing communications packet has
42  arrived
43
44.. index:: multiprocessing topologies
45
46Background
47==========
48
49RTEMS makes no assumptions regarding the connection media or topology of a
50multiprocessor system.  The tasks which compose a particular application can be
51spread among as many processors as needed to satisfy the application's timing
52requirements.  The application tasks can interact using a subset of the RTEMS
53directives as if they were on the same processor.  These directives allow
54application tasks to exchange data, communicate, and synchronize regardless of
55which processor they reside upon.
56
57The RTEMS multiprocessor execution model is multiple instruction streams with
58multiple data streams (MIMD).  This execution model has each of the processors
59executing code independent of the other processors.  Because of this
60parallelism, the application designer can more easily guarantee deterministic
61behavior.
62
63By supporting heterogeneous environments, RTEMS allows the systems designer to
64select the most efficient processor for each subsystem of the application.
65Configuring RTEMS for a heterogeneous environment is no more difficult than for
66a homogeneous one.  In keeping with RTEMS philosophy of providing transparent
67physical node boundaries, the minimal heterogeneous processing required is
68isolated in the MPCI layer.
69
70.. index:: nodes, definition
71
72Nodes
73-----
74
75A processor in a RTEMS system is referred to as a node.  Each node is assigned
76a unique non-zero node number by the application designer.  RTEMS assumes that
77node numbers are assigned consecutively from one to the ``maximum_nodes``
78configuration parameter.  The node number, node, and the maximum number of
79nodes, ``maximum_nodes``, in a system are found in the Multiprocessor
80Configuration Table.  The ``maximum_nodes`` field and the number of global
81objects, ``maximum_global_objects``, is required to be the same on all nodes in
82a system.
83
84The node number is used by RTEMS to identify each node when performing remote
85operations.  Thus, the Multiprocessor Communications Interface Layer (MPCI)
86must be able to route messages based on the node number.
87
88.. index:: global objects, definition
89
90Global Objects
91--------------
92
93All RTEMS objects which are created with the GLOBAL attribute will be known on
94all other nodes.  Global objects can be referenced from any node in the system,
95although certain directive specific restrictions (e.g. one cannot delete a
96remote object) may apply.  A task does not have to be global to perform
97operations involving remote objects.  The maximum number of global objects is
98the system is user configurable and can be found in the maximum_global_objects
99field in the Multiprocessor Configuration Table.  The distribution of tasks to
100processors is performed during the application design phase.  Dynamic task
101relocation is not supported by RTEMS.
102
103.. index:: global objects table
104
105Global Object Table
106-------------------
107
108RTEMS maintains two tables containing object information on every node in a
109multiprocessor system: a local object table and a global object table.  The
110local object table on each node is unique and contains information for all
111objects created on this node whether those objects are local or global.  The
112global object table contains information regarding all global objects in the
113system and, consequently, is the same on every node.
114
115Since each node must maintain an identical copy of the global object table, the
116maximum number of entries in each copy of the table must be the same.  The
117maximum number of entries in each copy is determined by the
118maximum_global_objects parameter in the Multiprocessor Configuration Table.
119This parameter, as well as the maximum_nodes parameter, is required to be the
120same on all nodes.  To maintain consistency among the table copies, every node
121in the system must be informed of the creation or deletion of a global object.
122
123.. index:: MPCI and remote operations
124
125Remote Operations
126-----------------
127
128When an application performs an operation on a remote global object, RTEMS must
129generate a Remote Request (RQ) message and send it to the appropriate node.
130After completing the requested operation, the remote node will build a Remote
131Response (RR) message and send it to the originating node.  Messages generated
132as a side-effect of a directive (such as deleting a global task) are known as
133Remote Processes (RP) and do not require the receiving node to respond.
134
135Other than taking slightly longer to execute directives on remote objects, the
136application is unaware of the location of the objects it acts upon.  The exact
137amount of overhead required for a remote operation is dependent on the media
138connecting the nodes and, to a lesser degree, on the efficiency of the
139user-provided MPCI routines.
140
141The following shows the typical transaction sequence during a remote
142application:
143
144#. The application issues a directive accessing a remote global object.
145
146#. RTEMS determines the node on which the object resides.
147
148#. RTEMS calls the user-provided MPCI routine ``GET_PACKET`` to obtain a packet
149   in which to build a RQ message.
150
151#. After building a message packet, RTEMS calls the user-provided MPCI routine
152   ``SEND_PACKET`` to transmit the packet to the node on which the object
153   resides (referred to as the destination node).
154
155#. The calling task is blocked until the RR message arrives, and control of the
156   processor is transferred to another task.
157
158#. The MPCI layer on the destination node senses the arrival of a packet
159   (commonly in an ISR), and calls the ``rtems_multiprocessing_announce``
160   directive.  This directive readies the Multiprocessing Server.
161
162#. The Multiprocessing Server calls the user-provided MPCI routine
163   ``RECEIVE_PACKET``, performs the requested operation, builds an RR message,
164   and returns it to the originating node.
165
166#. The MPCI layer on the originating node senses the arrival of a packet
167   (typically via an interrupt), and calls the RTEMS
168   ``rtems_multiprocessing_announce`` directive.  This directive readies the
169   Multiprocessing Server.
170
171#. The Multiprocessing Server calls the user-provided MPCI routine
172   ``RECEIVE_PACKET``, readies the original requesting task, and blocks until
173   another packet arrives.  Control is transferred to the original task which
174   then completes processing of the directive.
175
176If an uncorrectable error occurs in the user-provided MPCI layer, the fatal
177error handler should be invoked.  RTEMS assumes the reliable transmission and
178reception of messages by the MPCI and makes no attempt to detect or correct
179errors.
180
181.. index:: proxy, definition
182
183Proxies
184-------
185
186A proxy is an RTEMS data structure which resides on a remote node and is used
187to represent a task which must block as part of a remote operation. This action
188can occur as part of the ``rtems_semaphore_obtain`` and
189``rtems_message_queue_receive`` directives.  If the object were local, the
190task's control block would be available for modification to indicate it was
191blocking on a message queue or semaphore.  However, the task's control block
192resides only on the same node as the task.  As a result, the remote node must
193allocate a proxy to represent the task until it can be readied.
194
195The maximum number of proxies is defined in the Multiprocessor Configuration
196Table.  Each node in a multiprocessor system may require a different number of
197proxies to be configured.  The distribution of proxy control blocks is
198application dependent and is different from the distribution of tasks.
199
200Multiprocessor Configuration Table
201----------------------------------
202
203The Multiprocessor Configuration Table contains information needed by RTEMS
204when used in a multiprocessor system.  This table is discussed in detail in the
205section Multiprocessor Configuration Table of the Configuring a System chapter.
206
207Multiprocessor Communications Interface Layer
208=============================================
209
210The Multiprocessor Communications Interface Layer (MPCI) is a set of
211user-provided procedures which enable the nodes in a multiprocessor system to
212communicate with one another.  These routines are invoked by RTEMS at various
213times in the preparation and processing of remote requests.  Interrupts are
214enabled when an MPCI procedure is invoked.  It is assumed that if the execution
215mode and/or interrupt level are altered by the MPCI layer, that they will be
216restored prior to returning to RTEMS.
217
218.. index:: MPCI, definition
219
220The MPCI layer is responsible for managing a pool of buffers called packets and
221for sending these packets between system nodes.  Packet buffers contain the
222messages sent between the nodes.  Typically, the MPCI layer will encapsulate
223the packet within an envelope which contains the information needed by the MPCI
224layer.  The number of packets available is dependent on the MPCI layer
225implementation.
226
227.. index:: MPCI entry points
228
229The entry points to the routines in the user's MPCI layer should be placed in
230the Multiprocessor Communications Interface Table.  The user must provide entry
231points for each of the following table entries in a multiprocessor system:
232
233.. list-table::
234 :class: rtems-table
235
236 * - initialization
237   - initialize the MPCI
238 * - get_packet
239   - obtain a packet buffer
240 * - return_packet
241   - return a packet buffer
242 * - send_packet
243   - send a packet to another node
244 * - receive_packet
245   - called to get an arrived packet
246
247A packet is sent by RTEMS in each of the following situations:
248
249- an RQ is generated on an originating node;
250
251- an RR is generated on a destination node;
252
253- a global object is created;
254
255- a global object is deleted;
256
257- a local task blocked on a remote object is deleted;
258
259- during system initialization to check for system consistency.
260
261If the target hardware supports it, the arrival of a packet at a node may
262generate an interrupt.  Otherwise, the real-time clock ISR can check for the
263arrival of a packet.  In any case, the ``rtems_multiprocessing_announce``
264directive must be called to announce the arrival of a packet.  After exiting
265the ISR, control will be passed to the Multiprocessing Server to process the
266packet.  The Multiprocessing Server will call the get_packet entry to obtain a
267packet buffer and the receive_entry entry to copy the message into the buffer
268obtained.
269
270INITIALIZATION
271--------------
272
273The INITIALIZATION component of the user-provided MPCI layer is called as part
274of the ``rtems_initialize_executive`` directive to initialize the MPCI layer
275and associated hardware.  It is invoked immediately after all of the device
276drivers have been initialized.  This component should be adhere to the
277following prototype:
278
279.. index:: rtems_mpci_entry
280
281.. code-block:: c
282
283    rtems_mpci_entry user_mpci_initialization(
284        rtems_configuration_table *configuration
285    );
286
287where configuration is the address of the user's Configuration Table.
288Operations on global objects cannot be performed until this component is
289invoked.  The INITIALIZATION component is invoked only once in the life of any
290system.  If the MPCI layer cannot be successfully initialized, the fatal error
291manager should be invoked by this routine.
292
293One of the primary functions of the MPCI layer is to provide the executive with
294packet buffers.  The INITIALIZATION routine must create and initialize a pool
295of packet buffers.  There must be enough packet buffers so RTEMS can obtain one
296whenever needed.
297
298GET_PACKET
299----------
300
301The GET_PACKET component of the user-provided MPCI layer is called when RTEMS
302must obtain a packet buffer to send or broadcast a message.  This component
303should be adhere to the following prototype:
304
305.. code-block:: c
306
307    rtems_mpci_entry user_mpci_get_packet(
308        rtems_packet_prefix **packet
309    );
310
311where packet is the address of a pointer to a packet.  This routine always
312succeeds and, upon return, packet will contain the address of a packet.  If for
313any reason, a packet cannot be successfully obtained, then the fatal error
314manager should be invoked.
315
316RTEMS has been optimized to avoid the need for obtaining a packet each time a
317message is sent or broadcast.  For example, RTEMS sends response messages (RR)
318back to the originator in the same packet in which the request message (RQ)
319arrived.
320
321RETURN_PACKET
322-------------
323
324The RETURN_PACKET component of the user-provided MPCI layer is called when
325RTEMS needs to release a packet to the free packet buffer pool.  This component
326should be adhere to the following prototype:
327
328.. code-block:: c
329
330    rtems_mpci_entry user_mpci_return_packet(
331        rtems_packet_prefix *packet
332    );
333
334where packet is the address of a packet.  If the packet cannot be successfully
335returned, the fatal error manager should be invoked.
336
337RECEIVE_PACKET
338--------------
339
340The RECEIVE_PACKET component of the user-provided MPCI layer is called when
341RTEMS needs to obtain a packet which has previously arrived.  This component
342should be adhere to the following prototype:
343
344.. code-block:: c
345
346    rtems_mpci_entry user_mpci_receive_packet(
347        rtems_packet_prefix **packet
348    );
349
350where packet is a pointer to the address of a packet to place the message from
351another node.  If a message is available, then packet will contain the address
352of the message from another node.  If no messages are available, this entry
353packet should contain NULL.
354
355SEND_PACKET
356-----------
357
358The SEND_PACKET component of the user-provided MPCI layer is called when RTEMS
359needs to send a packet containing a message to another node.  This component
360should be adhere to the following prototype:
361
362.. code-block:: c
363
364    rtems_mpci_entry user_mpci_send_packet(
365        uint32_t               node,
366        rtems_packet_prefix  **packet
367    );
368
369where node is the node number of the destination and packet is the address of a
370packet which containing a message.  If the packet cannot be successfully sent,
371the fatal error manager should be invoked.
372
373If node is set to zero, the packet is to be broadcasted to all other nodes in
374the system.  Although some MPCI layers will be built upon hardware which
375support a broadcast mechanism, others may be required to generate a copy of the
376packet for each node in the system.
377
378.. COMMENT: XXX packet_prefix structure needs to be defined in this document
379
380Many MPCI layers use the ``packet_length`` field of the ``rtems_packet_prefix``
381portion of the packet to avoid sending unnecessary data.  This is especially
382useful if the media connecting the nodes is relatively slow.
383
384The ``to_convert`` field of the ``rtems_packet_prefix`` portion of the packet
385indicates how much of the packet in 32-bit units may require conversion in a
386heterogeneous system.
387
388.. index:: heterogeneous multiprocessing
389
390Supporting Heterogeneous Environments
391-------------------------------------
392
393Developing an MPCI layer for a heterogeneous system requires a thorough
394understanding of the differences between the processors which comprise the
395system.  One difficult problem is the varying data representation schemes used
396by different processor types.  The most pervasive data representation problem
397is the order of the bytes which compose a data entity.  Processors which place
398the least significant byte at the smallest address are classified as little
399endian processors.  Little endian byte-ordering is shown below:
400
401.. code-block:: c
402
403    +---------------+----------------+---------------+----------------+
404    |               |                |               |                |
405    |    Byte 3     |     Byte 2     |    Byte 1     |    Byte 0      |
406    |               |                |               |                |
407    +---------------+----------------+---------------+----------------+
408
409Conversely, processors which place the most significant byte at the smallest
410address are classified as big endian processors.  Big endian byte-ordering is
411shown below:
412
413.. code-block:: c
414
415    +---------------+----------------+---------------+----------------+
416    |               |                |               |                |
417    |    Byte 0     |     Byte 1     |    Byte 2     |    Byte 3      |
418    |               |                |               |                |
419    +---------------+----------------+---------------+----------------+
420
421Unfortunately, sharing a data structure between big endian and little endian
422processors requires translation into a common endian format.  An application
423designer typically chooses the common endian format to minimize conversion
424overhead.
425
426Another issue in the design of shared data structures is the alignment of data
427structure elements.  Alignment is both processor and compiler implementation
428dependent.  For example, some processors allow data elements to begin on any
429address boundary, while others impose restrictions.  Common restrictions are
430that data elements must begin on either an even address or on a long word
431boundary.  Violation of these restrictions may cause an exception or impose a
432performance penalty.
433
434Other issues which commonly impact the design of shared data structures include
435the representation of floating point numbers, bit fields, decimal data, and
436character strings.  In addition, the representation method for negative
437integers could be one's or two's complement.  These factors combine to increase
438the complexity of designing and manipulating data structures shared between
439processors.
440
441RTEMS addressed these issues in the design of the packets used to communicate
442between nodes.  The RTEMS packet format is designed to allow the MPCI layer to
443perform all necessary conversion without burdening the developer with the
444details of the RTEMS packet format.  As a result, the MPCI layer must be aware
445of the following:
446
447- All packets must begin on a four byte boundary.
448
449- Packets are composed of both RTEMS and application data.  All RTEMS data is
450  treated as 32-bit unsigned quantities and is in the first ``to_convert``
451  32-bit quantities of the packet.  The ``to_convert`` field is part of the
452  ``rtems_packet_prefix`` portion of the packet.
453
454- The RTEMS data component of the packet must be in native endian format.
455  Endian conversion may be performed by either the sending or receiving MPCI
456  layer.
457
458- RTEMS makes no assumptions regarding the application data component of the
459  packet.
460
461Operations
462==========
463
464Announcing a Packet
465-------------------
466
467The ``rtems_multiprocessing_announce`` directive is called by the MPCI layer to
468inform RTEMS that a packet has arrived from another node.  This directive can
469be called from an interrupt service routine or from within a polling routine.
470
471Directives
472==========
473
474This section details the additional directives required to support RTEMS in a
475multiprocessor configuration.  A subsection is dedicated to each of this
476manager's directives and describes the calling sequence, related constants,
477usage, and status codes.
478
479.. raw:: latex
480
481   \clearpage
482
483.. index:: announce arrival of package
484.. index:: rtems_multiprocessing_announce
485
486.. _rtems_multiprocessing_announce:
487
488MULTIPROCESSING_ANNOUNCE - Announce the arrival of a packet
489-----------------------------------------------------------
490
491CALLING SEQUENCE:
492    .. code-block:: c
493
494        void rtems_multiprocessing_announce( void );
495
496DIRECTIVE STATUS CODES:
497    NONE
498
499DESCRIPTION:
500    This directive informs RTEMS that a multiprocessing communications packet
501    has arrived from another node.  This directive is called by the
502    user-provided MPCI, and is only used in multiprocessor configurations.
503
504NOTES:
505    This directive is typically called from an ISR.
506
507    This directive will almost certainly cause the calling task to be
508    preempted.
509
510    This directive does not generate activity on remote nodes.
Note: See TracBrowser for help on using the repository browser.