source: rtems-docs/c-user/multiprocessing.rst @ 12dccfe

.. comment SPDX-License-Identifier: CC-BY-SA-4.0

.. Copyright (C) 1988, 2008 On-Line Applications Research Corporation (OAR)

.. index:: multiprocessing

Multiprocessing Manager
***********************

Introduction
============

In multiprocessor real-time systems, new requirements, such as sharing data and
global resources between processors, are introduced.  This requires an
efficient and reliable communications vehicle which allows all processors to
communicate with each other as necessary.  In addition, the ramifications of
multiple processors affect each and every characteristic of a real-time system,
almost always making them more complicated.

RTEMS addresses these issues by providing simple and flexible real-time
multiprocessing capabilities.  The executive easily lends itself to both
tightly-coupled and loosely-coupled configurations of the target system
hardware.  In addition, RTEMS supports systems composed of both homogeneous and
heterogeneous mixtures of processors and target boards.

A major design goal of the RTEMS executive was to transcend the physical
boundaries of the target hardware configuration.  This goal is achieved by
presenting the application software with a logical view of the target system
where the boundaries between processor nodes are transparent.  As a result, the
application developer may designate objects such as tasks, queues, events,
signals, semaphores, and memory blocks as global objects.  These global objects
may then be accessed by any task regardless of the physical location of the
object and the accessing task.  RTEMS automatically determines that the object
being accessed resides on another processor and performs the actions required
to access the desired object.  Simply stated, RTEMS allows the entire system,
both hardware and software, to be viewed logically as a single system.

The directives provided by the  Manager are:

- rtems_multiprocessing_announce_ - A multiprocessing communications packet has
  arrived

.. index:: multiprocessing topologies

Background
==========

RTEMS makes no assumptions regarding the connection media or topology of a
multiprocessor system.  The tasks which compose a particular application can be
spread among as many processors as needed to satisfy the application's timing
requirements.  The application tasks can interact using a subset of the RTEMS
directives as if they were on the same processor.  These directives allow
application tasks to exchange data, communicate, and synchronize regardless of
which processor they reside upon.

The RTEMS multiprocessor execution model is multiple instruction streams with
multiple data streams (MIMD).  This execution model has each of the processors
executing code independent of the other processors.  Because of this
parallelism, the application designer can more easily guarantee deterministic
behavior.

By supporting heterogeneous environments, RTEMS allows the systems designer to
select the most efficient processor for each subsystem of the application.
Configuring RTEMS for a heterogeneous environment is no more difficult than for
a homogeneous one.  In keeping with RTEMS philosophy of providing transparent
physical node boundaries, the minimal heterogeneous processing required is
isolated in the MPCI layer.

.. index:: nodes, definition

Nodes
-----

A processor in a RTEMS system is referred to as a node.  Each node is assigned
a unique non-zero node number by the application designer.  RTEMS assumes that
node numbers are assigned consecutively from one to the ``maximum_nodes``
configuration parameter.  The node number, node, and the maximum number of
nodes, ``maximum_nodes``, in a system are found in the Multiprocessor
Configuration Table.  The ``maximum_nodes`` field and the number of global
objects, ``maximum_global_objects``, is required to be the same on all nodes in
a system.

The node number is used by RTEMS to identify each node when performing remote
operations.  Thus, the Multiprocessor Communications Interface Layer (MPCI)
must be able to route messages based on the node number.

.. index:: global objects, definition

Global Objects
--------------

All RTEMS objects which are created with the GLOBAL attribute will be known on
all other nodes.  Global objects can be referenced from any node in the system,
although certain directive specific restrictions (e.g. one cannot delete a
remote object) may apply.  A task does not have to be global to perform
operations involving remote objects.  The maximum number of global objects is
the system is user configurable and can be found in the maximum_global_objects
field in the Multiprocessor Configuration Table.  The distribution of tasks to
processors is performed during the application design phase.  Dynamic task
relocation is not supported by RTEMS.

.. index:: global objects table

Global Object Table
-------------------

RTEMS maintains two tables containing object information on every node in a
multiprocessor system: a local object table and a global object table.  The
local object table on each node is unique and contains information for all
objects created on this node whether those objects are local or global.  The
global object table contains information regarding all global objects in the
system and, consequently, is the same on every node.

Since each node must maintain an identical copy of the global object table, the
maximum number of entries in each copy of the table must be the same.  The
maximum number of entries in each copy is determined by the
maximum_global_objects parameter in the Multiprocessor Configuration Table.
This parameter, as well as the maximum_nodes parameter, is required to be the
same on all nodes.  To maintain consistency among the table copies, every node
in the system must be informed of the creation or deletion of a global object.

.. index:: MPCI and remote operations

Remote Operations
-----------------

When an application performs an operation on a remote global object, RTEMS must
generate a Remote Request (RQ) message and send it to the appropriate node.
After completing the requested operation, the remote node will build a Remote
Response (RR) message and send it to the originating node.  Messages generated
as a side-effect of a directive (such as deleting a global task) are known as
Remote Processes (RP) and do not require the receiving node to respond.

Other than taking slightly longer to execute directives on remote objects, the
application is unaware of the location of the objects it acts upon.  The exact
amount of overhead required for a remote operation is dependent on the media
connecting the nodes and, to a lesser degree, on the efficiency of the
user-provided MPCI routines.

The following shows the typical transaction sequence during a remote
application:

#. The application issues a directive accessing a remote global object.

#. RTEMS determines the node on which the object resides.

#. RTEMS calls the user-provided MPCI routine ``GET_PACKET`` to obtain a packet
   in which to build a RQ message.

#. After building a message packet, RTEMS calls the user-provided MPCI routine
   ``SEND_PACKET`` to transmit the packet to the node on which the object
   resides (referred to as the destination node).

#. The calling task is blocked until the RR message arrives, and control of the
   processor is transferred to another task.

#. The MPCI layer on the destination node senses the arrival of a packet
   (commonly in an ISR), and calls the ``rtems_multiprocessing_announce``
   directive.  This directive readies the Multiprocessing Server.

#. The Multiprocessing Server calls the user-provided MPCI routine
   ``RECEIVE_PACKET``, performs the requested operation, builds an RR message,
   and returns it to the originating node.

#. The MPCI layer on the originating node senses the arrival of a packet
   (typically via an interrupt), and calls the RTEMS
   ``rtems_multiprocessing_announce`` directive.  This directive readies the
   Multiprocessing Server.

#. The Multiprocessing Server calls the user-provided MPCI routine
   ``RECEIVE_PACKET``, readies the original requesting task, and blocks until
   another packet arrives.  Control is transferred to the original task which
   then completes processing of the directive.

If an uncorrectable error occurs in the user-provided MPCI layer, the fatal
error handler should be invoked.  RTEMS assumes the reliable transmission and
reception of messages by the MPCI and makes no attempt to detect or correct
errors.

.. index:: proxy, definition

Proxies
-------

A proxy is an RTEMS data structure which resides on a remote node and is used
to represent a task which must block as part of a remote operation. This action
can occur as part of the ``rtems_semaphore_obtain`` and
``rtems_message_queue_receive`` directives.  If the object were local, the
task's control block would be available for modification to indicate it was
blocking on a message queue or semaphore.  However, the task's control block
resides only on the same node as the task.  As a result, the remote node must
allocate a proxy to represent the task until it can be readied.

The maximum number of proxies is defined in the Multiprocessor Configuration
Table.  Each node in a multiprocessor system may require a different number of
proxies to be configured.  The distribution of proxy control blocks is
application dependent and is different from the distribution of tasks.

Multiprocessor Configuration Table
----------------------------------

The Multiprocessor Configuration Table contains information needed by RTEMS
when used in a multiprocessor system.  This table is discussed in detail in the
section Multiprocessor Configuration Table of the Configuring a System chapter.

Multiprocessor Communications Interface Layer
=============================================

The Multiprocessor Communications Interface Layer (MPCI) is a set of
user-provided procedures which enable the nodes in a multiprocessor system to
communicate with one another.  These routines are invoked by RTEMS at various
times in the preparation and processing of remote requests.  Interrupts are
enabled when an MPCI procedure is invoked.  It is assumed that if the execution
mode and/or interrupt level are altered by the MPCI layer, that they will be
restored prior to returning to RTEMS.

.. index:: MPCI, definition

The MPCI layer is responsible for managing a pool of buffers called packets and
for sending these packets between system nodes.  Packet buffers contain the
messages sent between the nodes.  Typically, the MPCI layer will encapsulate
the packet within an envelope which contains the information needed by the MPCI
layer.  The number of packets available is dependent on the MPCI layer
implementation.

.. index:: MPCI entry points

The entry points to the routines in the user's MPCI layer should be placed in
the Multiprocessor Communications Interface Table.  The user must provide entry
points for each of the following table entries in a multiprocessor system:

.. list-table::
 :class: rtems-table

 * - initialization
   - initialize the MPCI
 * - get_packet
   - obtain a packet buffer
 * - return_packet
   - return a packet buffer
 * - send_packet
   - send a packet to another node
 * - receive_packet
   - called to get an arrived packet

A packet is sent by RTEMS in each of the following situations:

- an RQ is generated on an originating node;

- an RR is generated on a destination node;

- a global object is created;

- a global object is deleted;

- a local task blocked on a remote object is deleted;

- during system initialization to check for system consistency.

If the target hardware supports it, the arrival of a packet at a node may
generate an interrupt.  Otherwise, the real-time clock ISR can check for the
arrival of a packet.  In any case, the ``rtems_multiprocessing_announce``
directive must be called to announce the arrival of a packet.  After exiting
the ISR, control will be passed to the Multiprocessing Server to process the
packet.  The Multiprocessing Server will call the get_packet entry to obtain a
packet buffer and the receive_entry entry to copy the message into the buffer
obtained.

INITIALIZATION
--------------

The INITIALIZATION component of the user-provided MPCI layer is called as part
of the ``rtems_initialize_executive`` directive to initialize the MPCI layer
and associated hardware.  It is invoked immediately after all of the device
drivers have been initialized.  This component should be adhere to the
following prototype:

.. index:: rtems_mpci_entry

.. code-block:: c

    rtems_mpci_entry user_mpci_initialization(
        rtems_configuration_table *configuration
    );

where configuration is the address of the user's Configuration Table.
Operations on global objects cannot be performed until this component is
invoked.  The INITIALIZATION component is invoked only once in the life of any
system.  If the MPCI layer cannot be successfully initialized, the fatal error
manager should be invoked by this routine.

One of the primary functions of the MPCI layer is to provide the executive with
packet buffers.  The INITIALIZATION routine must create and initialize a pool
of packet buffers.  There must be enough packet buffers so RTEMS can obtain one
whenever needed.

GET_PACKET
----------

The GET_PACKET component of the user-provided MPCI layer is called when RTEMS
must obtain a packet buffer to send or broadcast a message.  This component
should be adhere to the following prototype:

.. code-block:: c

    rtems_mpci_entry user_mpci_get_packet(
        rtems_packet_prefix **packet
    );

where packet is the address of a pointer to a packet.  This routine always
succeeds and, upon return, packet will contain the address of a packet.  If for
any reason, a packet cannot be successfully obtained, then the fatal error
manager should be invoked.

RTEMS has been optimized to avoid the need for obtaining a packet each time a
message is sent or broadcast.  For example, RTEMS sends response messages (RR)
back to the originator in the same packet in which the request message (RQ)
arrived.

RETURN_PACKET
-------------

The RETURN_PACKET component of the user-provided MPCI layer is called when
RTEMS needs to release a packet to the free packet buffer pool.  This component
should be adhere to the following prototype:

.. code-block:: c

    rtems_mpci_entry user_mpci_return_packet(
        rtems_packet_prefix *packet
    );

where packet is the address of a packet.  If the packet cannot be successfully
returned, the fatal error manager should be invoked.

RECEIVE_PACKET
--------------

The RECEIVE_PACKET component of the user-provided MPCI layer is called when
RTEMS needs to obtain a packet which has previously arrived.  This component
should be adhere to the following prototype:

.. code-block:: c

    rtems_mpci_entry user_mpci_receive_packet(
        rtems_packet_prefix **packet
    );

where packet is a pointer to the address of a packet to place the message from
another node.  If a message is available, then packet will contain the address
of the message from another node.  If no messages are available, this entry
packet should contain NULL.

SEND_PACKET
-----------

The SEND_PACKET component of the user-provided MPCI layer is called when RTEMS
needs to send a packet containing a message to another node.  This component
should be adhere to the following prototype:

.. code-block:: c

    rtems_mpci_entry user_mpci_send_packet(
        uint32_t               node,
        rtems_packet_prefix  **packet
    );

where node is the node number of the destination and packet is the address of a
packet which containing a message.  If the packet cannot be successfully sent,
the fatal error manager should be invoked.

If node is set to zero, the packet is to be broadcasted to all other nodes in
the system.  Although some MPCI layers will be built upon hardware which
support a broadcast mechanism, others may be required to generate a copy of the
packet for each node in the system.

.. COMMENT: XXX packet_prefix structure needs to be defined in this document

Many MPCI layers use the ``packet_length`` field of the ``rtems_packet_prefix``
portion of the packet to avoid sending unnecessary data.  This is especially
useful if the media connecting the nodes is relatively slow.

The ``to_convert`` field of the ``rtems_packet_prefix`` portion of the packet
indicates how much of the packet in 32-bit units may require conversion in a
heterogeneous system.

.. index:: heterogeneous multiprocessing

Supporting Heterogeneous Environments
-------------------------------------

Developing an MPCI layer for a heterogeneous system requires a thorough
understanding of the differences between the processors which comprise the
system.  One difficult problem is the varying data representation schemes used
by different processor types.  The most pervasive data representation problem
is the order of the bytes which compose a data entity.  Processors which place
the least significant byte at the smallest address are classified as little
endian processors.  Little endian byte-ordering is shown below:

.. code-block:: c

    +---------------+----------------+---------------+----------------+
    |               |                |               |                |
    |    Byte 3     |     Byte 2     |    Byte 1     |    Byte 0      |
    |               |                |               |                |
    +---------------+----------------+---------------+----------------+

Conversely, processors which place the most significant byte at the smallest
address are classified as big endian processors.  Big endian byte-ordering is
shown below:

.. code-block:: c

    +---------------+----------------+---------------+----------------+
    |               |                |               |                |
    |    Byte 0     |     Byte 1     |    Byte 2     |    Byte 3      |
    |               |                |               |                |
    +---------------+----------------+---------------+----------------+

Unfortunately, sharing a data structure between big endian and little endian
processors requires translation into a common endian format.  An application
designer typically chooses the common endian format to minimize conversion
overhead.

Another issue in the design of shared data structures is the alignment of data
structure elements.  Alignment is both processor and compiler implementation
dependent.  For example, some processors allow data elements to begin on any
address boundary, while others impose restrictions.  Common restrictions are
that data elements must begin on either an even address or on a long word
boundary.  Violation of these restrictions may cause an exception or impose a
performance penalty.

Other issues which commonly impact the design of shared data structures include
the representation of floating point numbers, bit fields, decimal data, and
character strings.  In addition, the representation method for negative
integers could be one's or two's complement.  These factors combine to increase
the complexity of designing and manipulating data structures shared between
processors.

RTEMS addressed these issues in the design of the packets used to communicate
between nodes.  The RTEMS packet format is designed to allow the MPCI layer to
perform all necessary conversion without burdening the developer with the
details of the RTEMS packet format.  As a result, the MPCI layer must be aware
of the following:

- All packets must begin on a four byte boundary.

- Packets are composed of both RTEMS and application data.  All RTEMS data is
  treated as 32-bit unsigned quantities and is in the first ``to_convert``
  32-bit quantities of the packet.  The ``to_convert`` field is part of the
  ``rtems_packet_prefix`` portion of the packet.

- The RTEMS data component of the packet must be in native endian format.
  Endian conversion may be performed by either the sending or receiving MPCI
  layer.

- RTEMS makes no assumptions regarding the application data component of the
  packet.

Operations
==========

Announcing a Packet
-------------------

The ``rtems_multiprocessing_announce`` directive is called by the MPCI layer to
inform RTEMS that a packet has arrived from another node.  This directive can
be called from an interrupt service routine or from within a polling routine.

Directives
==========

This section details the additional directives required to support RTEMS in a
multiprocessor configuration.  A subsection is dedicated to each of this
manager's directives and describes the calling sequence, related constants,
usage, and status codes.

.. raw:: latex

   \clearpage

.. index:: announce arrival of package
.. index:: rtems_multiprocessing_announce

.. _rtems_multiprocessing_announce:

MULTIPROCESSING_ANNOUNCE - Announce the arrival of a packet
-----------------------------------------------------------

CALLING SEQUENCE:
    .. code-block:: c

        void rtems_multiprocessing_announce( void );

DIRECTIVE STATUS CODES:
    NONE

DESCRIPTION:
    This directive informs RTEMS that a multiprocessing communications packet
    has arrived from another node.  This directive is called by the
    user-provided MPCI, and is only used in multiprocessor configurations.

NOTES:
    This directive is typically called from an ISR.

    This directive will almost certainly cause the calling task to be
    preempted.

    This directive does not generate activity on remote nodes.