source: rtems-libbsd/libbsd.txt @ 43dc972

RTEMS BSD Library Guide
=======================
:toc:
:icons:
:numbered:
:website: http://www.rtems.org/

RTEMS uses FreeBSD 9.2 as the source of its TCP/IP and USB stacks.
This is a guide which captures information on the
process of merging code from FreeBSD, building this library,
RTEMS specific support files, and general guidelines on what
modifications to the FreeBSD source are permitted.

Goals of this effort are

* update TCP/IP and provide USB in RTEMS,
* ease updating to future FreeBSD versions,
* ease tracking changes in FreeBSD code,
* minimize manual changes in FreeBSD code, and
* define stable kernel/device driver API which is implemented
by both RTEMS and FreeBSD. This is the foundation of the port.

We will work to push our changes upstream to the FreeBSD Project
and minimize changes required at each update point.

*******************************************************************************
This is a work in progress and is very likely to be incomplete.
Please help by adding to it.
*******************************************************************************

== Getting Started

=== Tool Chain ===

You need a tool chain for RTEMS based on at least

* Binutils 2.24, and
* Newlib 2.1.0.

The Binutils version is required to ease the handling of linker command files.
The Newlib version is required since some standard files like `<sys/types.h>`
must be compatible enough for the files provided by the FreeBSD sources, e.g.
`<sys/socket.h>`.

=== Installation Overview ===

. You must configure your BSP with the +--disable-networking+ option to disable
the old network stack.  Make sure no header files of the old network stack are
installed.
. Clone the Git repository +git clone git://git.rtems.org/rtems-libbsd.git+.
. Change into the RTEMS BSD library root directory.
. Edit the `config.inc` Makefile configuration file and adjust it to your environment.
. Run +make clean+.
. Run +make install+.

=== Board Support Package Requirements ===

The RTEMS version must be at least 4.11.  The Board Support Package (BSP)
should support the
http://www.rtems.org/onlinedocs/doxygen/cpukit/html/group\__rtems\__interrupt__extension.html[Interrupt Manager Extension]
// The first underscores have to be masked to stop asciidoc interpreting them
to make use of generic FreeBSD based drivers.

The linker command file of the BSP must contain the following section
definitions:

-------------------------------------------------------------------------------
.rtemsroset : {
        KEEP (*(SORT(.rtemsroset.*)))
}

.rtemsrwset : {
        KEEP (*(SORT(.rtemsrwset.*)))
}
-------------------------------------------------------------------------------

The first output section can be placed in read-only memory.  The second output
section must be placed in read-write memory.  The output section name is not
relevant.  The output sections may also contain other input sections.

=== Board Support Package Configuration and Build ===

You need to configure RTEMS for the desired BSP and install it.  The BSP should
be configured with a disabled network stack.  The BSD library containing the
new network stack is a separate package.  Using a BSP installation containing
the old network stack may lead to confusion and unpredictable results.

The following script is used to build the `arm/realview_pbx_a9_qemu` BSP for
our internal testing purposes:

-------------------------------------------------------------------------------
#!/bin/sh

cd ${HOME}/sandbox
rm -rf b-realview_pbx_a9_qemu
mkdir b-realview_pbx_a9_qemu
cd b-realview_pbx_a9_qemu
${HOME}/git-rtems/configure \
        --prefix=${HOME}/sandbox/install \
        --target=arm-rtems4.11 \
        --enable-rtemsbsp=realview_pbx_a9_qemu \
        --disable-networking && \
        make && \
        make install
-------------------------------------------------------------------------------

The `arm/realview_pbx_a9_qemu` BSP running on the Qemu simulator has some
benefits for development and test of the BSD library

* it offers a NULL pointer read and write protection,
* Qemu is a fast simulator,
* Qemu provides support for GDB watchpoints,
* Qemu provides support for virtual Ethernet networks, e.g. TUN and bridge
devices (you can run multiple test instances on one virtual network).

=== BSD Library Configuration and Build ===

In the BSD library source directory edit the file `config.inc`.  Continuing on
the above, the `config.inc` used to match the above is:

-------------------------------------------------------------------------------
# Mandatory: Select your BSP and installation prefix
TARGET = arm-rtems4.11
BSP = realview_pbx_a9_qemu
PREFIX = $(HOME)/sandbox/install

# Optional: Separate installation base directory
INSTALL_BASE = $(PREFIX)/$(TARGET)/$(BSP)

# Optional: Network test configuration
TEST_RUNNER = $(BSP)
NET_CFG_SELF_IP = 10.0.0.2
NET_CFG_NETMASK = 255.255.0.0
NET_CFG_PEER_IP = 10.0.0.1
NET_CFG_GATEWAY_IP = 10.0.0.1
NET_TAP_INTERFACE = tap0
-------------------------------------------------------------------------------

Now you can build the BSD library and run the tests:

-------------------------------------------------------------------------------
make clean
make
make run_tests
-------------------------------------------------------------------------------

You can only run the tests directly in case a test runner is available.  The
following tests run without an external network.  It is strongly advised to run
them.

* commands01
* init01
* loopback01
* rwlock01
* selectpollkqueue01
* sleep01
* swi01
* syscalls01
* thread01
* timeout01
* unix01

To install the BSD library use this:

-------------------------------------------------------------------------------
make install
-------------------------------------------------------------------------------

=== BSD Library Initialization ===

Use the following code to initialize the BSD library:

-------------------------------------------------------------------------------
#include <assert.h>
#include <sysexits.h>

#include <machine/rtems-bsd-commands.h>
#include <rtems/bsd/bsd.h>

static void
network_ifconfig_lo0(void)
{
        int exit_code;
        char *lo0[] = {
                "ifconfig",
                "lo0",
                "inet",
                "127.0.0.1",
                "netmask",
                "255.255.255.0",
                NULL
        };
        char *lo0_inet6[] = {
                "ifconfig",
                "lo0",
                "inet6",
                "::1",
                "prefixlen",
                "128",
                NULL
        };

        exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0), lo0);
        assert(exit_code == EX_OK);

        exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0_inet6), lo0_inet6);
        assert(exit_code == EX_OK);
}

void
network_init(void)
{
        rtems_status_code sc;

        sc = rtems_bsd_initialize();
        assert(sc == RTEMS_SUCCESSFUL);

        network_ifconfig_lo0();
}
-------------------------------------------------------------------------------

This performs the basic network stack initialization with a loopback interface.
Further initialization must be done using the standard BSD network
configuration commands
http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)]
using `rtems_bsd_command_ifconfig()` and
http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)]
using `rtems_bsd_command_route()`.  For an example please have a look at
`testsuite/include/rtems/bsd/test/default-network-init.h`.

== Network Stack Features

http://roy.marples.name/projects/dhcpcd/index[DHCPCD(8)]:: DHCP client

https://developer.apple.com/library/mac/documentation/Networking/Reference/DNSServiceDiscovery_CRef/Reference/reference.html[dns_sd.h]:: DNS Service Discovery

http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-320.10/mDNSCore/mDNSEmbeddedAPI.h[mDNS]:: Multi-Cast DNS

http://www.freebsd.org/cgi/man.cgi?query=unix&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[UNIX(4)]:: UNIX-domain protocol family

http://www.freebsd.org/cgi/man.cgi?query=inet&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[INET(4)]:: Internet protocol family

http://www.freebsd.org/cgi/man.cgi?query=inet6&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[INET6(4)]:: Internet protocol version 6 family

http://www.freebsd.org/cgi/man.cgi?query=tcp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[TCP(4)]:: Internet Transmission Control Protocol

http://www.freebsd.org/cgi/man.cgi?query=udp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[UDP(4)]:: Internet User Datagram Protocol

http://www.freebsd.org/cgi/man.cgi?query=route&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[ROUTE(4)]:: Kernel packet forwarding database

http://www.freebsd.org/cgi/man.cgi?query=bpf&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[BPF(4)]:: Berkeley Packet Filter

http://www.freebsd.org/cgi/man.cgi?query=socket&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SOCKET(2)]:: Create an endpoint for communication

http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[KQUEUE(2)]:: Kernel event notification mechanism

http://www.freebsd.org/cgi/man.cgi?query=select&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SELECT(2)]:: Synchronous I/O multiplexing

http://www.freebsd.org/cgi/man.cgi?query=poll&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[POLL(2)]:: Synchronous I/O multiplexing

http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)]:: Manually manipulate the routing tables

http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)]:: Configure network interface parameters

http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=1&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[NETSTAT(1)]:: Show network status

http://www.freebsd.org/cgi/man.cgi?query=ping&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING(8)]:: Send ICMP ECHO_REQUEST packets to network hosts

http://www.freebsd.org/cgi/man.cgi?query=ping6&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING6(8)]:: Send ICMPv6 ECHO_REQUEST packets to network hosts

http://www.freebsd.org/cgi/man.cgi?query=sysctl&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[SYSCTL(3)]:: Get or set system information

http://www.freebsd.org/cgi/man.cgi?query=resolver&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[RESOLVER(3)]:: Resolver routines

http://www.freebsd.org/cgi/man.cgi?query=gethostbyname&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[GETHOSTBYNAME(3)]:: Get network host entry

== Network Interface Drivers

=== Link Up/Down Events

You can notifiy the application space of link up/down events in your network
interface driver via the if_link_state_change(LINK_STATE_UP/LINK_STATE_DOWN)
function.  The DHCPCD(8) client is a consumer of these events for example.
Make sure that the interface flag IFF_UP and the interface driver flag
IFF_DRV_RUNNING is set in case the link is up, otherwise ether_output() will
return the error status ENETDOWN.

== Issues and TODO

* Per-CPU data should be enabled once the new stack is ready for SMP.

* Per-CPU NETISR(9) should be enabled onece the new stack is ready for SMP.

* Multiple routing tables are not supported.  Every FIB value is set to zero
  (= BSD_DEFAULT_FIB).

* Process identifiers are not supported.  Every PID value is set to zero
  (= BSD_DEFAULT_PID).

* User credentials are not supported.  The following functions allow the
  operation for everyone
  - prison_equal_ip4(),
  - chgsbsize(),
  - cr_cansee(),
  - cr_canseesocket() and
  - cr_canseeinpcb().

* A basic USB functionality test that is known to work on Qemu is desirable.

* Adapt generic IRQ PIC interface code to Simple Vectored Interrupt Model
  so that those architectures can use new TCP/IP and USB code. 

* freebsd-userspace/rtems/include/sys/syslog.h is a copy from the old
  RTEMS TCP/IP stack. For some reason, the __printflike markers do not
  compile in this environment. We may want to use the FreeBSD syslog.h
  and get this addressed.

* in_cksum implementations for architectures not supported by FreeBSD.
  This will require figuring out where to put implementations that do
  not originate from FreeBSD and are populated via the script.

* MAC support functions are not thread-safe ("freebsd/lib/libc/posix1e/mac.c").

* IFCONFIG(8): IEEE80211 support is disabled.  This module depends on a XML
  parser and mmap().

* get_cyclecount(): The implementation is a security problem.

* What to do with the priority parameter present in the FreeBSD synchronization
  primitives and the thread creation functions?

* TASKQUEUE(9): Support spin mutexes.

* ZONE(9): Review allocator lock usage in rtems-bsd-chunk.c.

* KQUEUE(2): Choose proper lock for global kqueue list.

* TIMEOUT(9): Maybe use special task instead of timer server to call
  callout_tick().

* sysctl_handle_opaque(): Implement reliable snapshots.

* PING6(8): What to do with SIGALARM?

* <sys/param.h>: Update Newlib to use a MSIZE of 256.

* BPF(4): Add support for zero-copy buffers.

* UNIX(4): Fix race conditions in the area of socket object and file node
  destruction.  Add support for file descriptor transmission via control
  messages.

* PRINTF(9): Add support for log(), the %D format specifier is missing in the
  normal printf() family.

* Why is the interrupt server used?  The BSD interrupt handlers can block on
synchronization primitives like mutexes.  This is in contrast to RTEMS
interrupt service routines.  The BSPs using the generic interrupt support must
implement the `bsp_interrupt_vector_enable()` and
`bsp_interrupt_vector_disable()` routines.  They normally enable/disable a
particular interrupt source at the interrupt controller.  This can be used to
implement the interrupt server.  The interrupt server is a task that wakes-up
in case an associated interrupt happens.  The interrupt source is disabled in
a generic interrupt handler that wakes-up the interrupt server task.   Once the
postponed interrupt processing is performed in the interrupt server the
interrupt source is enabled again.

* Convert all BSP linkcmds to use a linkcmds.base so the sections are 
easier to insert.

* NIC Device Drivers
- Only common PCI NIC drivers have been included in the initial set. These
do not include any system on chip or ISA drivers.
- PCI configuration probe does not appear to happen to determine if a
NIC is in I/O or memory space. We have worked around this by using a
static hint to tell the fxp driver the correct mode. But this needs to
be addressed.
- The ISA drivers require more BSD infrastructure to be addressed. This was 
outside the scope of the initial porting effort.

== FreeBSD Source

You should be able to rely on FreebSD manual pages and documentation
for details on the code itself.

=== Automatically Generated FreeBSD Files

Some source and header files are automatically generated during the FreeBSD
build process.  The `Makefile.todo` file performs this manually.  The should be
included in `freebsd-to-rtems.py` script some time in the future.  For details,
see also
http://www.freebsd.org/cgi/man.cgi?query=kobj&sektion=9&apropos=0&manpath=FreeBSD+9.2-RELEASE[KOBJ(9)].

=== Rules for Modifying FreeBSD Source

Only add lines.  Subtract code by added `#ifndef __rtems__`.  This makes
merging easier in the future.  For example:

-------------------------------------------------------------------------------
/* Global variables for the kernel. */

#ifndef __rtems__
/* 1.1 */
extern char kernelname[MAXPATHLEN];
#endif /* __rtems__ */

extern int tick;                        /* usec per tick (1000000 / hz) */
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
#if defined(_KERNEL) || defined(_WANT_FILE)
#ifdef __rtems__
#include <rtems/libio_.h>
#include <sys/fcntl.h>
#endif /* __rtems__ */
/*
 * Kernel descriptor table.
 * One entry for each open kernel vnode and socket.
 *
 * Below is the list of locks that protects members in struct file.
 *
 * (f) protected with mtx_lock(mtx_pool_find(fp))
 * (d) cdevpriv_mtx
 * none not locked
 */
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
extern int profprocs;                   /* number of process's profiling */
#ifndef __rtems__
extern volatile int ticks;
#else /* __rtems__ */
#include <rtems/score/watchdogimpl.h>
#define ticks _Watchdog_Ticks_since_boot
#endif /* __rtems__ */

#endif /* _KERNEL */
-------------------------------------------------------------------------------

Add nothing (even blank lines) before or after the `__rtems__` guards.  Always
include a `__rtems__` in the guards to make searches easy.

== BSD Library Source

=== What is in the Git Repository

There is a self-contained kit with FreeBSD and RTEMS components pre-merged. The
Makefile in this kit is automatically generated.

Any changes to source in the `freebsd` directories will need to be merged
upstream into our master FreeBSD checkout, the `freebsd-org` submodule.

The repository contains two FreeBSD source trees.  In the `freebsd` directory
are the so called 'managed' FreeBSD sources used to build the BSD library.  The
FreeBSD source in `freebsd-org` is the 'master' version.  The
`freebsd-to-rtems.py` script is used to transfer files between the two trees.
In general terms, if you have modified managed FreeBSD sources, you will need
to run the script in 'revert' or 'reverse' mode using the `-R` switch.  This
will copy the source back to your local copy of the master FreeBSD source so
you can run `git diff` against the upstream FreeBSD source.  If you want to
transfer source files from the master FreeBSD source to the manged FreeBSD
sources, then you must run the script in 'forward' mode (the default).

=== Organization

The top level directory contains a few directories and files. The following
are important to understand

* `freebsd-to-rtems.py` - script to convert to and free FreeBSD and RTEMS trees,
* `Makefile` - automatically generated,
* `freebsd/` - from FreeBSD by script,
* `rtemsbsd/` - RTEMS specific implementations of FreeBSD kernel support routines,
* `testsuite/` - RTEMS specific tests, and
* `libbsd.txt` - documentation in Asciidoc.

== Moving Code Between Managed and Master FreeBSD Source

The script `freebsd-to-rtems.py` is used to copy code from FreeBSD to the
rtems-libbsd tree and to reverse this process. This script attempts to
automate this process as much as possible and performs some transformations
on the FreeBSD code. Its command line arguments are shown below:

----
freebsd-to-rtems.py [args]
  -?|-h|--help     print this and exit
  -d|--dry-run     run program but no modifications
  -D|--diff        provide diff of files between trees
  -e|--early-exit  evaluate arguments, print results, and exit
  -m|--makefile    just generate Makefile
  -R|--reverse     default FreeBSD -> RTEMS, reverse that
  -r|--rtems       RTEMS directory
  -f|--freebsd     FreeBSD directory
  -v|--verbose     enable verbose output mode
----

In its default mode of operation, freebsd-to-rtems.py is used to copy code
from FreeBSD to the rtems-libbsd tree and perform transformations.  In forward
mode, the script may be requested to just generate the Makefile.

In "reverse mode", this script undoes those transformations and copies 
the source code back to the FreeBSD SVN tree. This allows us to do 
'svn diff', evaluate changes made by the RTEMS Project, and report changes
back to FreeBSD upstream.

In either mode, the script may be asked to perform a dry-run or be verbose.
Also, in either mode, the script is also smart enough to avoid copying over
files which have not changed. This means that the timestamps of files are
not changed unless the contents change. The script will also report the
number of files which changed. In verbose mode, the script will print
the name of the files which are changed.

The following is an example forward run with no changes.

----
$ ~/newbsd/git/libbsd-8.2/freebsd-to-rtems.py \
    -r /home/joel/newbsd/git/libbsd-8.2 \
    -f /home/joel/newbsd/libbsd/freebsd-8.2 -v
Verbose:                yes
Dry Run:                no
Only Generate Makefile: no
RTEMS Directory:        /home/joel/newbsd/git/libbsd-8.2
FreeBSD Directory:      /home/joel/newbsd/libbsd/freebsd-8.2
Direction:              forward
Generating into /home/joel/newbsd/git/libbsd-8.2
0 files were changed.
----

The script may also be used to generate a diff in either forward or reverse
direction.

== Initialization of the BSD Library

The initialization of the BSD library is based on the FreeBSD SYSINIT(9)
infrastructure.  The key to initializing a system is to ensure that the desired
device drivers are explicitly pulled into the linked application.  This plus
linking against the BSD library (`libbsd.a`) will pull in the necessary FreeBSD
infrastructure.

The FreeBSD kernel is not a library like the RTEMS kernel.  It is a bunch of
object files linked together.  If we have a library, then creating the
executable is simple.  We begin with a start symbol and recursively resolve all
references.  With a bunch of object files linked together we need a different
mechanism.  Most object files don't know each other.  Lets say we have a driver
module.  The rest of the system has no references to this driver module.  The
driver module needs a way to tell the rest of the system: Hey, kernel I am
here, please use my services!

This registration of independent components is performed by SYSINIT(9) and
specializations:

http://www.freebsd.org/cgi/man.cgi?query=SYSINIT

The SYSINIT(9) uses some global data structures that are placed in a certain
section.  In the linker command file we need this:

-------------------------------------------------------------------------------
.rtemsroset : {
        KEEP (*(SORT(.rtemsroset.*)))
}

.rtemsrwset : {
        KEEP (*(SORT(.rtemsrwset.*)))
}
-------------------------------------------------------------------------------

This results for example in this executable layout:

-------------------------------------------------------------------------------
[...]
 *(SORT(.rtemsroset.*))
 .rtemsroset.bsd.modmetadata_set.begin
                0x000000000025fe00        0x0 libbsd.a(rtems-bsd-init.o)
                0x000000000025fe00                _bsd__start_set_modmetadata_set
 .rtemsroset.bsd.modmetadata_set.content
                0x000000000025fe00        0x8 libbsd.a(rtems-bsd-nexus.o)
 .rtemsroset.bsd.modmetadata_set.content
                0x000000000025fe08        0x4 libbsd.a(kern_module.o)
[...]
 .rtemsroset.bsd.modmetadata_set.content
                0x000000000025fe68        0x4 libbsd.a(mii.o)
 .rtemsroset.bsd.modmetadata_set.content
                0x000000000025fe6c        0x4 libbsd.a(mii_bitbang.o)
 .rtemsroset.bsd.modmetadata_set.end
                0x000000000025fe70        0x0 libbsd.a(rtems-bsd-init.o)
                0x000000000025fe70                _bsd__stop_set_modmetadata_set
[...]
.rtemsrwset     0x000000000030bad0      0x290
 *(SORT(.rtemsrwset.*))
 .rtemsrwset.bsd.sysinit_set.begin
                0x000000000030bad0        0x0 libbsd.a(rtems-bsd-init.o)
                0x000000000030bad0                _bsd__start_set_sysinit_set
 .rtemsrwset.bsd.sysinit_set.content
                0x000000000030bad0        0x4 libbsd.a(rtems-bsd-nexus.o)
 .rtemsrwset.bsd.sysinit_set.content
                0x000000000030bad4        0x8 libbsd.a(rtems-bsd-thread.o)
 .rtemsrwset.bsd.sysinit_set.content
                0x000000000030badc        0x4 libbsd.a(init_main.o)
[...]
 .rtemsrwset.bsd.sysinit_set.content
                0x000000000030bd54        0x4 libbsd.a(frag6.o)
 .rtemsrwset.bsd.sysinit_set.content
                0x000000000030bd58        0x8 libbsd.a(uipc_accf.o)
 .rtemsrwset.bsd.sysinit_set.end
                0x000000000030bd60        0x0 libbsd.a(rtems-bsd-init.o)
                0x000000000030bd60                _bsd__stop_set_sysinit_set
[...]
-------------------------------------------------------------------------------

Here you can see, that some global data structures are collected into
continuous memory areas.  This memory area can be identified by start and stop
symbols.  This constructs a table of uniform items.

The low level FreeBSD code calls at some time during the initialization the
mi_startup() function (machine independent startup).  This function will sort
the SYSINIT(9) set and call handler functions which perform further
initialization.  The last step is the scheduler invocation.

The SYSINIT(9) routines are run in mi_startup() which is called by
rtems_bsd_initialize().

This is also explained in "The Design and Implementation of the FreeBSD
Operating System" section 14.3 "Kernel Initialization".

In RTEMS we have a library and not a bunch of object files.  Thus we need a way
to pull-in the desired services out of the libbsd.  Here the
`rtems-bsd-sysinit.h` comes into play.  The SYSINIT(9) macros have been
modified and extended for RTEMS in `<sys/kernel.h>`:

-------------------------------------------------------------------------------
#ifndef __rtems__
#define C_SYSINIT(uniquifier, subsystem, order, func, ident)    \
        static struct sysinit uniquifier ## _sys_init = {       \
                subsystem,                                      \
                order,                                          \
                func,                                           \
                (ident)                                         \
        };                                                      \
        DATA_SET(sysinit_set,uniquifier ## _sys_init)
#else /* __rtems__ */
#define SYSINIT_ENTRY_NAME(uniquifier)                          \
        _bsd_ ## uniquifier ## _sys_init
#define SYSINIT_REFERENCE_NAME(uniquifier)                      \
        _bsd_ ## uniquifier ## _sys_init_ref
#define C_SYSINIT(uniquifier, subsystem, order, func, ident)    \
        struct sysinit SYSINIT_ENTRY_NAME(uniquifier) = {       \
                subsystem,                                      \
                order,                                          \
                func,                                           \
                (ident)                                         \
        };                                                      \
        RWDATA_SET(sysinit_set,SYSINIT_ENTRY_NAME(uniquifier))
#define SYSINIT_REFERENCE(uniquifier)                           \
        extern struct sysinit SYSINIT_ENTRY_NAME(uniquifier);   \
        static struct sysinit const * const                     \
        SYSINIT_REFERENCE_NAME(uniquifier) __used               \
        = &SYSINIT_ENTRY_NAME(uniquifier)
#define SYSINIT_MODULE_REFERENCE(mod)                           \
        SYSINIT_REFERENCE(mod ## module)
#define SYSINIT_DRIVER_REFERENCE(driver, bus)                   \
        SYSINIT_MODULE_REFERENCE(driver ## _ ## bus)
#define SYSINIT_DOMAIN_REFERENCE(dom)                           \
        SYSINIT_REFERENCE(domain_add_ ## dom)
#endif /* __rtems__ */
-------------------------------------------------------------------------------

Here you see that the SYSINIT(9) entries are no longer static.  The
\*_REFERENCE() macros will create references to the corresponding modules which
are later resolved by the linker.  The application has to provide an object
file with references to all required FreeBSD modules.

The FreeBSD device model is quite elaborated (with follow-ups):

http://www.freebsd.org/cgi/man.cgi?query=driver

The devices form a tree with the Nexus device at a high-level.  This Nexus
device is architecture specific in FreeBSD.  In RTEMS we have our own Nexus
device, see `rtemsbsd/bsp/bsp-bsd-nexus-devices.c`.

=== SYSCTL_NODE Example

During development, we had an undefined reference to
_bsd_sysctl__net_children that we had trouble tracking down. Thanks to
Chris Johns, we located it. He explained how to read SYSCTL_NODE
definitions. This line from freebsd/netinet/in_proto.c is attempting
to add the "inet" node to the parent node "_net".

----
SYSCTL_NODE(_net,      PF_INET,         inet,   CTLFLAG_RW, 0,
        "Internet Family");
----

Our problem was that we could not find where _bsd_sysctl__net_children
was defined. Chris suggested that when in doubt compile with -save-temps
and look at the preprocessed .i files. But he did not need that. He
explained that this the symbol name _bsd_sysctl__net_children was
automatically generated by a SYSCTL_NODE as follows:

* _bsd_ - added by RTEMS modifications to SYSCTL_NODE macro
* sysctl_ - boilerplace added by SYSCTL_NODE macro
* "" - empty string for parent node
* net - name of SYSCTL_NODE
* children - added by SYSCTL macros
  
This was all generated by a support macro declaring the node as this:

----
struct sysctl_oid_list SYSCTL_NODE_CHILDREN(parent, name);
----

Given this information, we located this SYSCTL_NODE declaration in 
kern/kern_mib.c

----
SYSCTL_NODE(, CTL_KERN,   kern,   CTLFLAG_RW, 0,
        "High kernel, proc, limits &c");
----

== Core FreeBSD APIs and RTEMS Replacements ==

=== SX(9) (Shared/exclusive locks) ===

http://www.freebsd.org/cgi/man.cgi?query=sx

Binary semaphores (this neglects the ability to allow shared access).

=== MUTEX(9) (Mutual exclusion) ===

http://www.freebsd.org/cgi/man.cgi?query=mutex

Binary semaphores (not recursive mutexes are not supported this way).

=== RWLOCK(9) (Reader/writer lock) ===

http://www.freebsd.org/cgi/man.cgi?query=rwlock

POSIX r/w lock.

=== RMLOCK(9) (Reader/writer lock optimized for mostly read access patterns) ===

Note:  This object was implemented as a wrapper for RWLOCK in the rm_lock header file.

http://www.freebsd.org/cgi/man.cgi?query=rmlock

POSIX r/w lock.

=== CONDVAR(9) (Condition variables) ===

http://www.freebsd.org/cgi/man.cgi?query=condvar

POSIX condition variables with modifications (hack).

=== CALLOUT(9) (Timer functions) ===

http://www.freebsd.org/cgi/man.cgi?query=callout

Timer server.

=== TASKQUEUE(9) (Asynchronous task execution) ===

http://www.freebsd.org/cgi/man.cgi?query=taskqueue

TBD.

=== KTHREAD(9), KPROC(9) (Tasks) ===

http://www.freebsd.org/cgi/man.cgi?query=kthread

http://www.freebsd.org/cgi/man.cgi?query=kproc

Tasks.

=== ZONE(9) (Zone allocator) ===

http://www.freebsd.org/cgi/man.cgi?query=zone

TBD.

=== devfs (Device file system) ===

Dummy, IMFS or new implementation (currently dummy).

=== psignal (Signals) ===

TBD.  Seems to be not needed.

=== poll, select ===

TBD.  Seems to be not needed.

=== RMAN(9) (Resource management) ===

http://www.freebsd.org/cgi/man.cgi?query=rman

TBD.  Seems to be not needed.

=== DEVCLASS(9), DEVICE(9), DRIVER(9), MAKE_DEV(9) (Device management) ===

http://www.freebsd.org/cgi/man.cgi?query=devclass

http://www.freebsd.org/cgi/man.cgi?query=device

http://www.freebsd.org/cgi/man.cgi?query=driver

http://www.freebsd.org/cgi/man.cgi?query=make_dev

Use FreeBSD implementation as far as possible.  FreeBSD has a nice API for
dynamic device handling.  It may be interesting for RTEMS to use this API
internally in the future.

=== BUS_SPACE(9), BUS_DMA(9) (Bus and DMA access) ===

http://www.freebsd.org/cgi/man.cgi?query=bus_space

http://www.freebsd.org/cgi/man.cgi?query=bus_dma

Likely BSP dependent.  A default implementation for memory mapped linear access
is easy to provide.  The current heap implementation supports all properties
demanded by bus_dma (including the boundary constraint).

== RTEMS Replacements by File Description ==

Note:  Files with a status of USB are used by the USB test and have at least
been partially tested.  If they contain both USB and Nic, then they are used
by both and MAY contain methods that have not been tested yet.  Files that
are only used by the Nic test are the most suspect.

----
rtems-libbsd File:      rtems-bsd-assert.c
FreeBSD File:           rtems-bsd-config.h redefines BSD_ASSERT.
Description:            This file contains the support method rtems_bsd_assert_func().
Status:                 USB, Nic

rtems-libbsd File:      rtems-bsd-autoconf.c
FreeBSD File:           FreeBSD has BSP specific autoconf.c 
Description:            This file contains configuration methods that are used to setup the system.
Status:                 USB 

rtems-libbsd File:      rtems-bsd-bus-dma.c
FreeBSD File:           FreeBSD has BSP specific busdma_machdep.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-bus-dma-mbuf.c        
FreeBSD File:           FreeBSD has BSP specific busdma_machdep.c
Description:            
Status:                 Nic

rtems-libbsd File:      rtems-bsd-callout.c              
FreeBSD File:           kern/kern_timeout.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-cam.c
FreeBSD File:           cam/cam_sim.c
Description:            
Status:                 USB 

rtems-libbsd File:      rtems-bsd-condvar.c              
FreeBSD File:           kern/kern_condvar.c
Description:            
Status:                 USB 

rtems-libbsd File:      rtems-bsd-copyinout.c
FreeBSD File:           bsp specific copyinout.c )
Description:            Note: The FreeBSD file is split with some methods being in rtems-bsd-support
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-delay.c
FreeBSD File:           bsp specific file with multiple names
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-descrip.c 
FreeBSD File:           kern/kern_descrip.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-generic.c             
FreeBSD File:           kern/sys_generic.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-init.c 
FreeBSD File:           N/A
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-init-with-irq.c
FreeBSD File:           N/A
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-jail.c
FreeBSD File:           kern/kern_jail.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-lock.c
FreeBSD File:           kern/subr_lock.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-log.c         
FreeBSD File:           kern/subr_prf.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-malloc.c
FreeBSD File:           kern/kern_malloc.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-mutex.c
FreeBSD File:           kern/kern_mutex.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-newproc.c
FreeBSD File:           N/A
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-nexus.c
FreeBSD File:           bsp specific nexus.c
Description:            
Status:                 USB 

rtems-libbsd File:      rtems-bsd-panic.c                
FreeBSD File:           boot/common/panic.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-rwlock.c              
FreeBSD File:           kern_rwlock.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-shell.c               
FreeBSD File:           N/A
Description:            
Status:                 USB 

rtems-libbsd File:      rtems-bsd-signal.c              
FreeBSD File:           kern/kern_sig.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-smp.c                 
FreeBSD File:           N/A
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-support.c             
FreeBSD File:           bsp specific copyinout.c 
Description:            Note: the FreeBSD file is split with some methods being in rtems-bsd-copyinout.
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-sx.c                  
FreeBSD File:           kern/kern_sx.c
Description:            Status: USB, Nic 

rtems-libbsd File:      rtems-bsd-synch.c               
FreeBSD File:           kern/kern_synch.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-syscalls.c            
FreeBSD File:           User API for kern/uipc_syscalls.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-sysctlbyname.c        
FreeBSD File:           User API for sysctlbyname(3)
Description:            
Status: 

rtems-libbsd File:      rtems-bsd-sysctl.c              
FreeBSD File:           User API for sysctl(8)
Description:            
Status: 

rtems-libbsd File:      rtems-bsd-sysctlnametomib.c     
FreeBSD File:           User API for sysctlnametomib
Description:            
Status: 

rtems-libbsd File:      rtems-bsd-taskqueue.c           
FreeBSD File:           kern/subr_taskqueue.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-thread.c                      
FreeBSD File:           kern/kern_kthread.c
Description:            
Status:                 USB, Nic 

rtems-libbsd File:      rtems-bsd-timeout.c             
FreeBSD File:           kern/kern_timeout.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-timesupport.c         
FreeBSD File:           kern/kern_clock.c
Description:            
Status:                 Nic 

rtems-libbsd File:      rtems-bsd-vm_glue.c             
FreeBSD File:           vm/vm_glue.c
Description:            
Status:                 USB, Nic 
----

== Notes by File ==

altq_subr.c - Arbitrary choices were made in this file that RTEMS would
not support tsc frequency change.  Additionally, the clock frequency
for machclk_freq is always measured for RTEMS.

conf.h - In order to add make_dev and destroy_dev, variables in the cdev
structure that were not being used were conditionally compiled out. The
capability of supporting children did not appear to be needed and was
not implemented in the rtems version of these routines.
 
== NICs Status ==

----
Driver                  Symbol                          Status
======                  ======                          ======
RealTek                 _bsd_re_pcimodule_sys_init      Links 
EtherExpress            _bsd_fxp_pcimodule_sys_init     Links
DEC tulip               _bsd_dc_pcimodule_sys_init      Links
Broadcom BCM57xxx       _bsd_bce_pcimodule_sys_init     Links
Broadcom BCM4401        _bsd_bfe_pcimodule_sys_init     Links
Broadcom BCM570x        _bsd_bge_pcimodule_sys_init     Needs Symbols (A)
E1000 IGB               _bsd_igb_pcimodule_sys_init     Links
E1000 EM                _bsd_em_pcimodule_sys_init      Links
----


Symbols (A)
         pci_get_vpd_ident
 
== Problems to report to FreeBSD ==

The MMAP_NOT_AVAILABLE define is inverted on its usage.  When it is
defined the mmap method is called. Additionally, it is not used 
thoroughly. It is not used in the unmap portion of the source.
The file rec_open.c uses the define MMAP_NOT_AVAILABLE to wrap
the call to mmap and file rec_close.c uses the munmap method.