RTEMS BSD Library Guide ======================= :toc: :icons: :numbered: :website: http://www.rtems.org/ RTEMS uses FreeBSD 9.2 as the source of its TCP/IP and USB stacks. This is a guide which captures information on the process of merging code from FreeBSD, building this library, RTEMS specific support files, and general guidelines on what modifications to the FreeBSD source are permitted. Goals of this effort are * update TCP/IP and provide USB in RTEMS, * ease updating to future FreeBSD versions, * ease tracking changes in FreeBSD code, * minimize manual changes in FreeBSD code, and * define stable kernel/device driver API which is implemented by both RTEMS and FreeBSD. This is the foundation of the port. We will work to push our changes upstream to the FreeBSD Project and minimize changes required at each update point. ******************************************************************************* This is a work in progress and is very likely to be incomplete. Please help by adding to it. ******************************************************************************* == Getting Started === Tool Chain === You need a tool chain for RTEMS based on at least RSB 4.12 April 2016 or later. === Installation Overview === . You must configure your BSP with the +--disable-networking+ option to disable the old network stack. Make sure no header files of the old network stack are installed. . Clone the Git repository +git clone git://git.rtems.org/rtems-libbsd.git+. . Change into the RTEMS BSD library root directory. . Edit the `config.inc` configuration file and adjust it to your environment. . Run +waf configure ...+. . Run +waf+. . Run +waf install+. Refer to the README.waf for Waf building instructions. Make sure the submodules have been initialised and are updated. If a 'git status' says `rtems_waf` need updating run the submodule update command: $ git submodule rtems_waf update === Board Support Package Requirements === The RTEMS version must be at least 4.12. The Board Support Package (BSP) should support the http://www.rtems.org/onlinedocs/doxygen/cpukit/html/group\__rtems\__interrupt__extension.html[Interrupt Manager Extension] // The first underscores have to be masked to stop asciidoc interpreting them to make use of generic FreeBSD based drivers. The linker command file of the BSP must contain the following section definitions: ------------------------------------------------------------------------------- .rtemsroset : { KEEP (*(SORT(.rtemsroset.*))) } .rtemsrwset : { KEEP (*(SORT(.rtemsrwset.*))) } ------------------------------------------------------------------------------- The first output section can be placed in read-only memory. The second output section must be placed in read-write memory. The output section name is not relevant. The output sections may also contain other input sections. === Board Support Package Configuration and Build === You need to configure RTEMS for the desired BSP and install it. The BSP should be configured with a disabled network stack. The BSD library containing the new network stack is a separate package. Using a BSP installation containing the old network stack may lead to confusion and unpredictable results. The following script is used to build the `arm/xilinx_zynq_a9_qemu` BSP for our internal testing purposes: ------------------------------------------------------------------------------- #!/bin/sh cd ${HOME}/sandbox rm -rf b-xilinx_zynq_a9_qemu mkdir b-xilinx_zynq_a9_qemu cd b-xilinx_zynq_a9_qemu ${HOME}/git-rtems/configure \ --prefix=${HOME}/sandbox/install \ --target=arm-rtems4.12 \ --enable-rtemsbsp=xilinx_zynq_a9_qemu \ --disable-networking && \ make && \ make install ------------------------------------------------------------------------------- The `arm/xilinx_zynq_a9_qemu` BSP running on the Qemu simulator has some benefits for development and test of the BSD library * it offers a NULL pointer read and write protection, * Qemu is a fast simulator, * Qemu provides support for GDB watchpoints, * Qemu provides support for virtual Ethernet networks, e.g. TUN and bridge devices (you can run multiple test instances on one virtual network). === BSD Library Configuration and Build === The build system based on the Waf build system. To build with Waf please refer to the README.waf file. ===== Example Configuration ===== In the BSD library source directory edit the file `config.inc`. Continuing on the above, the `config.inc` used to match the above is: ------------------------------------------------------------------------------- # Mandatory: Select your BSP and installation prefix TARGET = arm-rtems4.12 BSP = xilinx_zynq_a9_qemu PREFIX = $(HOME)/sandbox/install # Optional: Separate installation base directory INSTALL_BASE = $(PREFIX)/$(TARGET)/$(BSP) # Optional: Network test configuration TEST_RUNNER = $(BSP) NET_CFG_SELF_IP = 10.0.0.2 NET_CFG_NETMASK = 255.255.0.0 NET_CFG_PEER_IP = 10.0.0.1 NET_CFG_GATEWAY_IP = 10.0.0.1 NET_TAP_INTERFACE = tap0 ------------------------------------------------------------------------------- === BSD Library Initialization === To initialise the BSD Library create a suitable rc.conf file. The FreeBSD man page rc.conf(5) provides the details needed to create a suitable format file: https://www.freebsd.org/cgi/man.cgi?rc.conf You can call one of three functions to run the initialisation once BSD has initialised: - rtems_bsd_run_etc_rc_conf: Run /etc/rc.conf. - rtems_bsd_run_rc_conf: Run a user supplied file. - rtems_bsd_run_rc_conf_script: Run the in memory line feed separated text string. For exapmle: void network_init(void) { rtems_status_code sc; sc = rtems_bsd_initialize(); assert(sc == RTEMS_SUCCESSFUL); rtems_bsd_run_etc_rc_conf(true); /* verbose = true */ } By default the networking support is builtin. Other directives can be added and are found in 'machine/rtems-bsd-rc-conf-directives.h'. Please check the file for the list. The following network names are supported: cloned_interfaces ifconfig_'interface' defaultrouter hostname For example: # # My BSD initialisation. # hostname="myhost" cloned_interfaces="vlan0 vlan1" ifconfig_re0="inet inet 10.10.10.10 netmask 255.255.255.0" fconfig_vlan0="inet 10.11.10.10 255.255.255.0 vlan 101 vlandev re0" defaultrouter="10.10.10.1" You can also intialise the BSD library using code. The following code to initialize the BSD library: ------------------------------------------------------------------------------- #include #include #include #include static void network_ifconfig_lo0(void) { int exit_code; char *lo0[] = { "ifconfig", "lo0", "inet", "127.0.0.1", "netmask", "255.255.255.0", NULL }; char *lo0_inet6[] = { "ifconfig", "lo0", "inet6", "::1", "prefixlen", "128", NULL }; exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0), lo0); assert(exit_code == EX_OK); exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0_inet6), lo0_inet6); assert(exit_code == EX_OK); } void network_init(void) { rtems_status_code sc; sc = rtems_bsd_initialize(); assert(sc == RTEMS_SUCCESSFUL); network_ifconfig_lo0(); } ------------------------------------------------------------------------------- This performs the basic network stack initialization with a loopback interface. Further initialization must be done using the standard BSD network configuration commands http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)] using `rtems_bsd_command_ifconfig()` and http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)] using `rtems_bsd_command_route()`. For an example please have a look at `testsuite/include/rtems/bsd/test/default-network-init.h`. === Task Priorities and Stack Size === The default task priority is 96 for the interrupt server task (name "IRQS"), 98 for the timer server task (name "TIME") and 100 for all other tasks. The application may provide their own implementation of the `rtems_bsd_get_task_priority()` function (for example in the module which calls `rtems_bsd_initialize()`) if different values are desired. The task stack size is determined by the `rtems_bsd_get_task_stack_size()` function which may be provided by the application in case the default is not appropriate. === Size for Allocator Domains === The size for an allocator domain can be specified via the `rtems_bsd_get_allocator_domain_size()` function. The application may provide their own implementation of the `rtems_bsd_get_allocator_domain_size()` function (for example in the module which calls `rtems_bsd_initialize()`) if different values are desired. The default size is 8MiB for all domains. == Network Stack Features http://roy.marples.name/projects/dhcpcd/index[DHCPCD(8)]:: DHCP client https://developer.apple.com/library/mac/documentation/Networking/Reference/DNSServiceDiscovery_CRef/Reference/reference.html[dns_sd.h]:: DNS Service Discovery http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-320.10/mDNSCore/mDNSEmbeddedAPI.h[mDNS]:: Multi-Cast DNS http://www.freebsd.org/cgi/man.cgi?query=unix&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[UNIX(4)]:: UNIX-domain protocol family http://www.freebsd.org/cgi/man.cgi?query=inet&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[INET(4)]:: Internet protocol family http://www.freebsd.org/cgi/man.cgi?query=inet6&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[INET6(4)]:: Internet protocol version 6 family http://www.freebsd.org/cgi/man.cgi?query=tcp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[TCP(4)]:: Internet Transmission Control Protocol http://www.freebsd.org/cgi/man.cgi?query=udp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[UDP(4)]:: Internet User Datagram Protocol http://www.freebsd.org/cgi/man.cgi?query=route&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[ROUTE(4)]:: Kernel packet forwarding database http://www.freebsd.org/cgi/man.cgi?query=bpf&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[BPF(4)]:: Berkeley Packet Filter http://www.freebsd.org/cgi/man.cgi?query=socket&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SOCKET(2)]:: Create an endpoint for communication http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[KQUEUE(2)]:: Kernel event notification mechanism http://www.freebsd.org/cgi/man.cgi?query=select&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SELECT(2)]:: Synchronous I/O multiplexing http://www.freebsd.org/cgi/man.cgi?query=poll&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[POLL(2)]:: Synchronous I/O multiplexing http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)]:: Manually manipulate the routing tables http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)]:: Configure network interface parameters http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=1&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[NETSTAT(1)]:: Show network status http://www.freebsd.org/cgi/man.cgi?query=ping&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING(8)]:: Send ICMP ECHO_REQUEST packets to network hosts http://www.freebsd.org/cgi/man.cgi?query=ping6&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING6(8)]:: Send ICMPv6 ECHO_REQUEST packets to network hosts http://www.freebsd.org/cgi/man.cgi?query=sysctl&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[SYSCTL(3)]:: Get or set system information http://www.freebsd.org/cgi/man.cgi?query=resolver&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[RESOLVER(3)]:: Resolver routines http://www.freebsd.org/cgi/man.cgi?query=gethostbyname&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[GETHOSTBYNAME(3)]:: Get network host entry == Network Interface Drivers === Link Up/Down Events You can notifiy the application space of link up/down events in your network interface driver via the if_link_state_change(LINK_STATE_UP/LINK_STATE_DOWN) function. The DHCPCD(8) client is a consumer of these events for example. Make sure that the interface flag IFF_UP and the interface driver flag IFF_DRV_RUNNING is set in case the link is up, otherwise ether_output() will return the error status ENETDOWN. == Shell Commands === HOSTNAME(1) In addition to the standard options the RTEMS version of the HOSTNAME(1) command supports the -m flag to set/get the multicast hostname of the mDNS resolver instance. See also rtems_mdns_sethostname() and rtems_mdns_gethostname(). == Qemu Use the following script to set up a virtual network with three tap devices connected via one bridge device. ------------------------------------------------------------------------------- #!/bin/sh -x user=`whoami` interfaces=(1 2 3) tap=qtap bri=qbri case $1 in up) sudo -i brctl addbr $bri for i in ${interfaces[@]} ; do sudo -i tunctl -t $tap$i -u $user ; sudo -i ifconfig $tap$i up ; sudo -i brctl addif $bri $tap$i ; done sudo -i ifconfig $bri up ;; down) for i in ${interfaces[@]} ; do sudo -i ifconfig $tap$i down ; sudo -i tunctl -d $tap$i ; done sudo -i ifconfig $bri down sudo -i brctl delbr $bri ;; esac ------------------------------------------------------------------------------- Connect your Qemu instance to one of the tap devices, e.g. ------------------------------------------------------------------------------- qemu-system-i386 -m 512 -boot a -cpu pentium3 \ -drive file=$HOME/qemu/pc386_fda,index=0,if=floppy,format=raw \ -drive file=fat:$HOME/qemu/hd,format=raw \ -net nic,model=e1000,macaddr=0e:b0:ba:5e:ba:11 \ -net tap,ifname=qtap1,script=no,downscript=no \ -nodefaults -nographic -serial stdio ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- qemu-system-arm \ -serial null \ -serial mon:stdio \ -nographic \ -M xilinx-zynq-a9 \ -net nic,model=cadence_gem,macaddr=0e:b0:ba:5e:ba:11 \ -net tap,ifname=qtap1,script=no,downscript=no \ -m 256M \ -kernel build/arm-rtems4.12-xilinx_zynq_a9_qemu/media01.exe ------------------------------------------------------------------------------- Make sure that each Qemu instance uses its own MAC address to avoid an address conflict (or otherwise use it as a test). To connect the Qemu instances with your local network use the following (replace 'eth0' with the network interface of your host). ------------------------------------------------------------------------------- ifconfig eth0 0.0.0.0 brctl addif qbri eth0 dhclient qbri ------------------------------------------------------------------------------- === VDE and QEMU On FreeBSD you can create VDE or the Virtual Distributed Ethernet to create a network environment that does not need to run qemu as root or needing to drop the tap's privileges to run qemu. VDE creates a software switch with a default of 32 ports which means a single kernel tap can support 32 qemu networking sessions. To use VDE you need to build qemu with VDE support. The RSB can detect a VDE plug and enable VDE support in qemu when building. On FreeBSD install the VDE support with: # pkg install -u vde2 Build qemu with the RSB. To network create a bridge and a tap. The network is 10.10.1.0/24. On FreeBSD add to your /etc/rc.conf: cloned_interfaces="bridge0 tap0" autobridge_interfaces="bridge0" autobridge_bridge0="re0 tap0" ifconfig_re0="up" ifconfig_tap0="up" ifconfig_bridge0="inet 10.1.1.2 netmask 255.255.255.0" defaultrouter="10.10.1.1" Start the VDE switch as root: # sysctl net.link.tap.user_open=1 # sysctl net.link.tap.up_on_open=1 # vde_switch -d -s /tmp/vde1 -M /tmp/mgmt1 -tap tap0 -m 660 --mgmtmode 660 # chmod 660 /dev/tap0 You can connect to the VDE switch's management channel using: $ vdeterm /tmp/mgmt1 To run qemu: $ qemu-system-arm \ -serial null \ -serial mon:stdio \ -nographic \ -M xilinx-zynq-a9 \ -net nic,model=cadence_gem,macaddr=0e:b0:ba:5e:ba:11 \ -net vde,id=vde0,sock=/tmp/vde1 -m 256M \ -kernel build/arm-rtems4.12-xilinx_zynq_a9_qemu/rcconf02.exe == Issues and TODO * PCI support on x86 uses a quick and dirty hack, see pci_reserve_map(). * Priority queues are broken with clustered scheduling. * Per-CPU data should be enabled once the new stack is ready for SMP. * Per-CPU NETISR(9) should be enabled onece the new stack is ready for SMP. * Multiple routing tables are not supported. Every FIB value is set to zero (= BSD_DEFAULT_FIB). * Process identifiers are not supported. Every PID value is set to zero (= BSD_DEFAULT_PID). * User credentials are not supported. The following functions allow the operation for everyone - prison_equal_ip4(), - chgsbsize(), - cr_cansee(), - cr_canseesocket() and - cr_canseeinpcb(). * A basic USB functionality test that is known to work on Qemu is desirable. * Adapt generic IRQ PIC interface code to Simple Vectored Interrupt Model so that those architectures can use new TCP/IP and USB code. * freebsd-userspace/rtems/include/sys/syslog.h is a copy from the old RTEMS TCP/IP stack. For some reason, the __printflike markers do not compile in this environment. We may want to use the FreeBSD syslog.h and get this addressed. * in_cksum implementations for architectures not supported by FreeBSD. This will require figuring out where to put implementations that do not originate from FreeBSD and are populated via the script. * MAC support functions are not thread-safe ("freebsd/lib/libc/posix1e/mac.c"). * IFCONFIG(8): IEEE80211 support is disabled. This module depends on a XML parser and mmap(). * get_cyclecount(): The implementation is a security problem. * What to do with the priority parameter present in the FreeBSD synchronization primitives and the thread creation functions? * TASKQUEUE(9): Support spin mutexes. * ZONE(9): Review allocator lock usage in rtems-bsd-chunk.c. * KQUEUE(2): Choose proper lock for global kqueue list. * TIMEOUT(9): Maybe use special task instead of timer server to call callout_tick(). * sysctl_handle_opaque(): Implement reliable snapshots. * PING6(8): What to do with SIGALARM? * : Update Newlib to use a MSIZE of 256. * BPF(4): Add support for zero-copy buffers. * UNIX(4): Fix race conditions in the area of socket object and file node destruction. Add support for file descriptor transmission via control messages. * PRINTF(9): Add support for log(), the %D format specifier is missing in the normal printf() family. * Why is the interrupt server used? The BSD interrupt handlers can block on synchronization primitives like mutexes. This is in contrast to RTEMS interrupt service routines. The BSPs using the generic interrupt support must implement the `bsp_interrupt_vector_enable()` and `bsp_interrupt_vector_disable()` routines. They normally enable/disable a particular interrupt source at the interrupt controller. This can be used to implement the interrupt server. The interrupt server is a task that wakes-up in case an associated interrupt happens. The interrupt source is disabled in a generic interrupt handler that wakes-up the interrupt server task. Once the postponed interrupt processing is performed in the interrupt server the interrupt source is enabled again. * Convert all BSP linkcmds to use a linkcmds.base so the sections are easier to insert. * NIC Device Drivers - Only common PCI NIC drivers have been included in the initial set. These do not include any system on chip or ISA drivers. - PCI configuration probe does not appear to happen to determine if a NIC is in I/O or memory space. We have worked around this by using a static hint to tell the fxp driver the correct mode. But this needs to be addressed. - The ISA drivers require more BSD infrastructure to be addressed. This was outside the scope of the initial porting effort. == FreeBSD Source You should be able to rely on FreebSD manual pages and documentation for details on the code itself. === Automatically Generated FreeBSD Files Some source and header files are automatically generated during the FreeBSD build process. The `Makefile.todo` file performs this manually. The should be included in `freebsd-to-rtems.py` script some time in the future. For details, see also http://www.freebsd.org/cgi/man.cgi?query=kobj&sektion=9&apropos=0&manpath=FreeBSD+9.2-RELEASE[KOBJ(9)]. === Rules for Modifying FreeBSD Source Only add lines. If your patch contains lines starting with a '-', then this is wrong. Subtract code by added `#ifndef __rtems__`. This makes merging easier in the future. For example: ------------------------------------------------------------------------------- /* Global variables for the kernel. */ #ifndef __rtems__ /* 1.1 */ extern char kernelname[MAXPATHLEN]; #endif /* __rtems__ */ extern int tick; /* usec per tick (1000000 / hz) */ ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- #if defined(_KERNEL) || defined(_WANT_FILE) #ifdef __rtems__ #include #include #endif /* __rtems__ */ /* * Kernel descriptor table. * One entry for each open kernel vnode and socket. * * Below is the list of locks that protects members in struct file. * * (f) protected with mtx_lock(mtx_pool_find(fp)) * (d) cdevpriv_mtx * none not locked */ ------------------------------------------------------------------------------- ------------------------------------------------------------------------------- extern int profprocs; /* number of process's profiling */ #ifndef __rtems__ extern volatile int ticks; #else /* __rtems__ */ #include #define ticks _Watchdog_Ticks_since_boot #endif /* __rtems__ */ #endif /* _KERNEL */ ------------------------------------------------------------------------------- Add nothing (even blank lines) before or after the `__rtems__` guards. Always include a `__rtems__` in the guards to make searches easy, so use * `#ifndef __rtems__`, * `#ifdef __rtems__`, * `#else /* __rtems__ */`, and * `#endif /* __rtems__ */`. The guards must start at the begin of the line. Examples for wrong guards: ------------------------------------------------------------------------------- static void guards_must_start_at_the_begin_of_the_line(int j) { #ifdef __rtems__ return (j + 1); #else /* __rtems__ */ return (j + 2); #endif /* __rtems__ */ } static void missing_rtems_comments_in_the_guards(int j) { #ifdef __rtems__ return (j + 3); #else return (j + 4); #endif } ------------------------------------------------------------------------------- The FreeBSD build and configuration system uses option header files, e.g. `#include "opt_xyz.h"` in an unmodified FreeBSD file. This include is transformed by the import script into `#include `. Do not disable option header includes via guards. Instead, add an empty option header, e.g. `touch rtemsbsd/include/rtems/bsd/local/opt_xyz.h`. ------------------------------------------------------------------------------- /* WRONG */ #ifndef __rtems__ #include #endif /* __rtems__ */ ------------------------------------------------------------------------------- In general, provide empty header files and do not guard includes. For new code use http://www.freebsd.org/cgi/man.cgi?query=style&apropos=0&sektion=9&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[STYLE(9)]. Do not format original FreeBSD code. == BSD Library Source === What is in the Git Repository There is a self-contained kit with FreeBSD and RTEMS components pre-merged. The Waf wscript in this kit is automatically generated. Any changes to source in the `freebsd` directories will need to be merged upstream into our master FreeBSD checkout, the `freebsd-org` submodule. The repository contains two FreeBSD source trees. In the `freebsd` directory are the so called 'managed' FreeBSD sources used to build the BSD library. The FreeBSD source in `freebsd-org` is the 'master' version. The `freebsd-to-rtems.py` script is used to transfer files between the two trees. In general terms, if you have modified managed FreeBSD sources, you will need to run the script in 'revert' or 'reverse' mode using the `-R` switch. This will copy the source back to your local copy of the master FreeBSD source so you can run `git diff` against the upstream FreeBSD source. If you want to transfer source files from the master FreeBSD source to the manged FreeBSD sources, then you must run the script in 'forward' mode (the default). === Organization The top level directory contains a few directories and files. The following are important to understand * `freebsd-to-rtems.py` - script to convert to and free FreeBSD and RTEMS trees, * `create-kernel-namespace.sh` - script to create the kernel namespace header RTEMS, reverse that -r|--rtems RTEMS Libbsd directory (default: '.') -f|--freebsd FreeBSD SVN directory (default: 'freebsd-org') -v|--verbose enable verbose output mode ---- In its default mode of operation, freebsd-to-rtems.py is used to copy code from FreeBSD to the rtems-libbsd tree and perform transformations. In forward mode, the script may be requested to just generate the Waf script. In "reverse mode", this script undoes those transformations and copies the source code back to the FreeBSD SVN tree. This allows us to do 'svn diff', evaluate changes made by the RTEMS Project, and report changes back to FreeBSD upstream. In either mode, the script may be asked to perform a dry-run or be verbose. Also, in either mode, the script is also smart enough to avoid copying over files which have not changed. This means that the timestamps of files are not changed unless the contents change. The script will also report the number of files which changed. In verbose mode, the script will print the name of the files which are changed. To add or update files in the RTEMS FreeBSD tree first run the 'reverse mode' and move the current set of patches FreeBSD. The script may warn you if a file is not present at the destination for the direction. This can happen as files not avaliable at the FreeBSD snapshot point have been specially added to the RTEMS FreeBSD tree. Warnings can also appear if you have changed the list of files in libbsd.py. The reverse mode will result in the FreeBSD having uncommitted changes. You can ignore these. Once the reverse process has finished edit libbsd.py and add any new files then run the forwad mode to bring those files into the RTEMS FreeBSD tree. The following is an example forward run with no changes. ---- $ ~/newbsd/git/libbsd-8.2/freebsd-to-rtems.py \ -r /home/joel/newbsd/git/libbsd-8.2 \ -f /home/joel/newbsd/libbsd/freebsd-8.2 -v Verbose: yes (1) Dry Run: no Only Generate Makefile: no RTEMS Directory: /home/joel/newbsd/git/libbsd-8.2 FreeBSD Directory: /home/joel/newbsd/libbsd/freebsd-8.2 Direction: forward Generating into /home/joel/newbsd/git/libbsd-8.2 0 files were changed. ---- The script may also be used to generate a diff in either forward or reverse direction. You can add more than one verbose option (-v) to the command line and get more detail and debug level information from the command. == FreeBSD version of imported files and directories . *, trunk, 2017-01-09, 1f8e4a995a6ede4bdb24e6d335ccda2bdb0175ab. == How to import code from FreeBSD . In case you import files from a special FreeBSD version, then update the list above. . Run `git status` and make sure your working directory is clean. . Run `./freebsd-to-rtems.py -R` . Run `./freebsd-to-rtems.py` . Run `git status` and make sure your working directory is clean. If you see modified files, then the `freebsd-to-rtems.py` script needs to be fixed first. . Add the files to import to `libbsd.py`. . Run `./freebsd-to-rtems.py` . Immediately check in the imported files without the changes to `libbsd_waf.py`. Do not touch the imported files yourself at this point. . Port the imported files to RTEMS. See 'Rules for Modifying FreeBSD Source'. . Add a test to the testsuite if possible. . Run `./create-kernel-namespace.sh` if you imported kernel space headers. Add only your new defines via `git add -p rtemsbsd/include/machine/rtems-bsd-kernel-namespace.h`. . Create one commit from this. The -S or --stats option generates reports the changes we have made to FreeBSD. If the code has been reserved into the original FreeBSD tree it will show nothing has changed. To see what we have change: $ cd freebsd-org $ git checkout -- . $ cd .. $ ./freebsd-to-rtems.py -R -S -d The report lists the files change based on the opacity level. The opacity is a measure on how much of a file differs from the original FreeBSD source. The lower the value the more transparent the source file it. == Porting of userspace utilities The theory behind the described method is to put all BSS and initialized data objects into a named section. This section then will be saved before the code is executed and restored after it has finished. This method limits to a single threaded execution of the application but minimizes the necessary changes to the original FreeBSD code. . Import and commit the unchanged source files like described above. . Add the files to the libbsd.py and build them. . Check the sources for everything that can be made const. This type of patches should go back to the upstream FreeBSD sources. . Move static variables out of functions if necessary (search for "static"). These patches most likely will not be accepted into FreeBSD. . Add a rtems_bsd_command_PROGNAME() wrapper function to the source file containing the main function (e.g. PROGNAME = pfctl). For an example look at `rtems_bsd_command_pfctl()` in `freebsd/sbin/pfctl/pfctl.c`. . You probably have to use getopt_r() instead of getopt(). Have a look at `freebsd/sbin/pfctl/pfctl.c`. . Build the libbsd without optimization. . Use the `userspace-header-gen.py` to generate some necessary header files. It will generate one `rtems-bsd-PROGNAME-MODULE-data.h` per object file, one `rtems-bsd-PROGNAME-namespace.h` and one `rtems-bsd-PROGNAME-data.h`. To call the script, you have to compile the objects and afterwards run the helper script with a call similar to this one: `python ./userspace-header-gen.py build/arm-rtems4.12-xilinx_zynq_a9_qemu/freebsd/sbin/pfctl/*.o -p pfctl` Replace the name (given via -p option) by the name of the userspace tool. It has to match the name that is used in the RTEMS linker set further below. . If you regenerated files that have already been generated, you may have to remove RTEMS-specific names from the namespace. The defaults (linker set names and rtems_bsd_program_xxx) should already be filtered. . Put the generated header files into the same folder like the source files. . Include `PROGNAME-rtems-bsd-namespace.h` at the top of each source file and the `PROGNAME-rtems-bsd-MODULE-data.h` after the include section of the corresponding source files. . Include `machine/rtems-bsd-program.h` at the top of the include block in each source file. . Create one compilable commit. == Initialization of the BSD Library The initialization of the BSD library is based on the FreeBSD SYSINIT(9) infrastructure. The key to initializing a system is to ensure that the desired device drivers are explicitly pulled into the linked application. This plus linking against the BSD library (`libbsd.a`) will pull in the necessary FreeBSD infrastructure. The FreeBSD kernel is not a library like the RTEMS kernel. It is a bunch of object files linked together. If we have a library, then creating the executable is simple. We begin with a start symbol and recursively resolve all references. With a bunch of object files linked together we need a different mechanism. Most object files don't know each other. Lets say we have a driver module. The rest of the system has no references to this driver module. The driver module needs a way to tell the rest of the system: Hey, kernel I am here, please use my services! This registration of independent components is performed by SYSINIT(9) and specializations: http://www.freebsd.org/cgi/man.cgi?query=SYSINIT The SYSINIT(9) uses some global data structures that are placed in a certain section. In the linker command file we need this: ------------------------------------------------------------------------------- .rtemsroset : { KEEP (*(SORT(.rtemsroset.*))) } .rtemsrwset : { KEEP (*(SORT(.rtemsrwset.*))) } ------------------------------------------------------------------------------- This results for example in this executable layout: ------------------------------------------------------------------------------- [...] *(SORT(.rtemsroset.*)) .rtemsroset.bsd.modmetadata_set.begin 0x000000000025fe00 0x0 libbsd.a(rtems-bsd-init.o) 0x000000000025fe00 _bsd__start_set_modmetadata_set .rtemsroset.bsd.modmetadata_set.content 0x000000000025fe00 0x8 libbsd.a(rtems-bsd-nexus.o) .rtemsroset.bsd.modmetadata_set.content 0x000000000025fe08 0x4 libbsd.a(kern_module.o) [...] .rtemsroset.bsd.modmetadata_set.content 0x000000000025fe68 0x4 libbsd.a(mii.o) .rtemsroset.bsd.modmetadata_set.content 0x000000000025fe6c 0x4 libbsd.a(mii_bitbang.o) .rtemsroset.bsd.modmetadata_set.end 0x000000000025fe70 0x0 libbsd.a(rtems-bsd-init.o) 0x000000000025fe70 _bsd__stop_set_modmetadata_set [...] .rtemsrwset 0x000000000030bad0 0x290 *(SORT(.rtemsrwset.*)) .rtemsrwset.bsd.sysinit_set.begin 0x000000000030bad0 0x0 libbsd.a(rtems-bsd-init.o) 0x000000000030bad0 _bsd__start_set_sysinit_set .rtemsrwset.bsd.sysinit_set.content 0x000000000030bad0 0x4 libbsd.a(rtems-bsd-nexus.o) .rtemsrwset.bsd.sysinit_set.content 0x000000000030bad4 0x8 libbsd.a(rtems-bsd-thread.o) .rtemsrwset.bsd.sysinit_set.content 0x000000000030badc 0x4 libbsd.a(init_main.o) [...] .rtemsrwset.bsd.sysinit_set.content 0x000000000030bd54 0x4 libbsd.a(frag6.o) .rtemsrwset.bsd.sysinit_set.content 0x000000000030bd58 0x8 libbsd.a(uipc_accf.o) .rtemsrwset.bsd.sysinit_set.end 0x000000000030bd60 0x0 libbsd.a(rtems-bsd-init.o) 0x000000000030bd60 _bsd__stop_set_sysinit_set [...] ------------------------------------------------------------------------------- Here you can see, that some global data structures are collected into continuous memory areas. This memory area can be identified by start and stop symbols. This constructs a table of uniform items. The low level FreeBSD code calls at some time during the initialization the mi_startup() function (machine independent startup). This function will sort the SYSINIT(9) set and call handler functions which perform further initialization. The last step is the scheduler invocation. The SYSINIT(9) routines are run in mi_startup() which is called by rtems_bsd_initialize(). This is also explained in "The Design and Implementation of the FreeBSD Operating System" section 14.3 "Kernel Initialization". In RTEMS we have a library and not a bunch of object files. Thus we need a way to pull-in the desired services out of the libbsd. Here the `rtems-bsd-sysinit.h` comes into play. The SYSINIT(9) macros have been modified and extended for RTEMS in ``: ------------------------------------------------------------------------------- #ifndef __rtems__ #define C_SYSINIT(uniquifier, subsystem, order, func, ident) \ static struct sysinit uniquifier ## _sys_init = { \ subsystem, \ order, \ func, \ (ident) \ }; \ DATA_SET(sysinit_set,uniquifier ## _sys_init) #else /* __rtems__ */ #define SYSINIT_ENTRY_NAME(uniquifier) \ _bsd_ ## uniquifier ## _sys_init #define SYSINIT_REFERENCE_NAME(uniquifier) \ _bsd_ ## uniquifier ## _sys_init_ref #define C_SYSINIT(uniquifier, subsystem, order, func, ident) \ struct sysinit SYSINIT_ENTRY_NAME(uniquifier) = { \ subsystem, \ order, \ func, \ (ident) \ }; \ RWDATA_SET(sysinit_set,SYSINIT_ENTRY_NAME(uniquifier)) #define SYSINIT_REFERENCE(uniquifier) \ extern struct sysinit SYSINIT_ENTRY_NAME(uniquifier); \ static struct sysinit const * const \ SYSINIT_REFERENCE_NAME(uniquifier) __used \ = &SYSINIT_ENTRY_NAME(uniquifier) #define SYSINIT_MODULE_REFERENCE(mod) \ SYSINIT_REFERENCE(mod ## module) #define SYSINIT_DRIVER_REFERENCE(driver, bus) \ SYSINIT_MODULE_REFERENCE(driver ## _ ## bus) #define SYSINIT_DOMAIN_REFERENCE(dom) \ SYSINIT_REFERENCE(domain_add_ ## dom) #endif /* __rtems__ */ ------------------------------------------------------------------------------- Here you see that the SYSINIT(9) entries are no longer static. The \*_REFERENCE() macros will create references to the corresponding modules which are later resolved by the linker. The application has to provide an object file with references to all required FreeBSD modules. The FreeBSD device model is quite elaborated (with follow-ups): http://www.freebsd.org/cgi/man.cgi?query=driver The devices form a tree with the Nexus device at a high-level. This Nexus device is architecture specific in FreeBSD. In RTEMS we have our own Nexus device, see `rtemsbsd/bsp/bsp-bsd-nexus-devices.c`. === SYSCTL_NODE Example During development, we had an undefined reference to _bsd_sysctl__net_children that we had trouble tracking down. Thanks to Chris Johns, we located it. He explained how to read SYSCTL_NODE definitions. This line from freebsd/netinet/in_proto.c is attempting to add the "inet" node to the parent node "_net". ---- SYSCTL_NODE(_net, PF_INET, inet, CTLFLAG_RW, 0, "Internet Family"); ---- Our problem was that we could not find where _bsd_sysctl__net_children was defined. Chris suggested that when in doubt compile with -save-temps and look at the preprocessed .i files. But he did not need that. He explained that this the symbol name _bsd_sysctl__net_children was automatically generated by a SYSCTL_NODE as follows: * _bsd_ - added by RTEMS modifications to SYSCTL_NODE macro * sysctl_ - boilerplace added by SYSCTL_NODE macro * "" - empty string for parent node * net - name of SYSCTL_NODE * children - added by SYSCTL macros This was all generated by a support macro declaring the node as this: ---- struct sysctl_oid_list SYSCTL_NODE_CHILDREN(parent, name); ---- Given this information, we located this SYSCTL_NODE declaration in kern/kern_mib.c ---- SYSCTL_NODE(, CTL_KERN, kern, CTLFLAG_RW, 0, "High kernel, proc, limits &c"); ---- == Core FreeBSD APIs and RTEMS Replacements == === SX(9) (Shared/exclusive locks) === http://www.freebsd.org/cgi/man.cgi?query=sx Binary semaphores (this neglects the ability to allow shared access). === MUTEX(9) (Mutual exclusion) === http://www.freebsd.org/cgi/man.cgi?query=mutex Binary semaphores (not recursive mutexes are not supported this way). === RWLOCK(9) (Reader/writer lock) === http://www.freebsd.org/cgi/man.cgi?query=rwlock POSIX r/w lock. === RMLOCK(9) (Reader/writer lock optimized for mostly read access patterns) === Note: This object was implemented as a wrapper for RWLOCK in the rm_lock header file. http://www.freebsd.org/cgi/man.cgi?query=rmlock POSIX r/w lock. === CONDVAR(9) (Condition variables) === http://www.freebsd.org/cgi/man.cgi?query=condvar POSIX condition variables with modifications (hack). === CALLOUT(9) (Timer functions) === http://www.freebsd.org/cgi/man.cgi?query=callout Timer server. === TASKQUEUE(9) (Asynchronous task execution) === http://www.freebsd.org/cgi/man.cgi?query=taskqueue TBD. === KTHREAD(9), KPROC(9) (Tasks) === http://www.freebsd.org/cgi/man.cgi?query=kthread http://www.freebsd.org/cgi/man.cgi?query=kproc Tasks. === ZONE(9) (Zone allocator) === http://www.freebsd.org/cgi/man.cgi?query=zone TBD. === devfs (Device file system) === There is a minimal implementation based on IMFS. The mount point is fixed to "/dev". Note that the devfs is only used by the cdev subsystem. cdev has been adapted so that the full path (including the leading "/dev") is given to devfs. This saves some copy operations. devfs_create() first creates the full path and then creates an IMFS generic node for the device. TBD: remove empty paths on devfs_destroy(). === psignal (Signals) === TBD. Seems to be not needed. === poll, select === TBD. Seems to be not needed. === RMAN(9) (Resource management) === http://www.freebsd.org/cgi/man.cgi?query=rman TBD. Seems to be not needed. === DEVCLASS(9), DEVICE(9), DRIVER(9), MAKE_DEV(9) (Device management) === http://www.freebsd.org/cgi/man.cgi?query=devclass http://www.freebsd.org/cgi/man.cgi?query=device http://www.freebsd.org/cgi/man.cgi?query=driver http://www.freebsd.org/cgi/man.cgi?query=make_dev Use FreeBSD implementation as far as possible. FreeBSD has a nice API for dynamic device handling. It may be interesting for RTEMS to use this API internally in the future. === BUS_SPACE(9), BUS_DMA(9) (Bus and DMA access) === http://www.freebsd.org/cgi/man.cgi?query=bus_space http://www.freebsd.org/cgi/man.cgi?query=bus_dma Likely BSP dependent. A default implementation for memory mapped linear access is easy to provide. The current heap implementation supports all properties demanded by bus_dma (including the boundary constraint). == RTEMS Replacements by File Description == Note: Files with a status of USB are used by the USB test and have at least been partially tested. If they contain both USB and Nic, then they are used by both and MAY contain methods that have not been tested yet. Files that are only used by the Nic test are the most suspect. ---- rtems-libbsd File: rtems-bsd-assert.c FreeBSD File: rtems-bsd-config.h redefines BSD_ASSERT. Description: This file contains the support method rtems_bsd_assert_func(). Status: USB, Nic rtems-libbsd File: rtems-bsd-autoconf.c FreeBSD File: FreeBSD has BSP specific autoconf.c Description: This file contains configuration methods that are used to setup the system. Status: USB rtems-libbsd File: rtems-bsd-bus-dma.c FreeBSD File: FreeBSD has BSP specific busdma_machdep.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-bus-dma-mbuf.c FreeBSD File: FreeBSD has BSP specific busdma_machdep.c Description: Status: Nic rtems-libbsd File: rtems-bsd-callout.c FreeBSD File: kern/kern_timeout.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-cam.c FreeBSD File: cam/cam_sim.c Description: Status: USB rtems-libbsd File: rtems-bsd-condvar.c FreeBSD File: kern/kern_condvar.c Description: Status: USB rtems-libbsd File: rtems-bsd-copyinout.c FreeBSD File: bsp specific copyinout.c ) Description: Note: The FreeBSD file is split with some methods being in rtems-bsd-support Status: Nic rtems-libbsd File: rtems-bsd-delay.c FreeBSD File: bsp specific file with multiple names Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-descrip.c FreeBSD File: kern/kern_descrip.c Description: Status: Nic rtems-libbsd File: rtems-bsd-generic.c FreeBSD File: kern/sys_generic.c Description: Status: Nic rtems-libbsd File: rtems-bsd-init.c FreeBSD File: N/A Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-init-with-irq.c FreeBSD File: N/A Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-jail.c FreeBSD File: kern/kern_jail.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-lock.c FreeBSD File: kern/subr_lock.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-log.c FreeBSD File: kern/subr_prf.c Description: Status: Nic rtems-libbsd File: rtems-bsd-malloc.c FreeBSD File: kern/kern_malloc.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-mutex.c FreeBSD File: kern/kern_mutex.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-newproc.c FreeBSD File: N/A Description: Status: Nic rtems-libbsd File: rtems-bsd-nexus.c FreeBSD File: bsp specific nexus.c Description: Status: USB rtems-libbsd File: rtems-bsd-panic.c FreeBSD File: boot/common/panic.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-rwlock.c FreeBSD File: kern_rwlock.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-shell.c FreeBSD File: N/A Description: Status: USB rtems-libbsd File: rtems-bsd-signal.c FreeBSD File: kern/kern_sig.c Description: Status: Nic rtems-libbsd File: rtems-bsd-smp.c FreeBSD File: N/A Description: Status: Nic rtems-libbsd File: rtems-bsd-support.c FreeBSD File: bsp specific copyinout.c Description: Note: the FreeBSD file is split with some methods being in rtems-bsd-copyinout. Status: USB, Nic rtems-libbsd File: rtems-bsd-sx.c FreeBSD File: kern/kern_sx.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-synch.c FreeBSD File: kern/kern_synch.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-syscalls.c FreeBSD File: User API for kern/uipc_syscalls.c Description: Status: Nic rtems-libbsd File: rtems-bsd-sysctlbyname.c FreeBSD File: User API for sysctlbyname(3) Description: Status: rtems-libbsd File: rtems-bsd-sysctl.c FreeBSD File: User API for sysctl(8) Description: Status: rtems-libbsd File: rtems-bsd-sysctlnametomib.c FreeBSD File: User API for sysctlnametomib Description: Status: rtems-libbsd File: rtems-bsd-taskqueue.c FreeBSD File: kern/subr_taskqueue.c Description: Status: Nic rtems-libbsd File: rtems-bsd-thread.c FreeBSD File: kern/kern_kthread.c Description: Status: USB, Nic rtems-libbsd File: rtems-bsd-timeout.c FreeBSD File: kern/kern_timeout.c Description: Status: Nic rtems-libbsd File: rtems-bsd-timesupport.c FreeBSD File: kern/kern_clock.c Description: Status: Nic rtems-libbsd File: rtems-bsd-vm_glue.c FreeBSD File: vm/vm_glue.c Description: Status: USB, Nic ---- == Notes by File == altq_subr.c - Arbitrary choices were made in this file that RTEMS would not support tsc frequency change. Additionally, the clock frequency for machclk_freq is always measured for RTEMS. conf.h - In order to add make_dev and destroy_dev, variables in the cdev structure that were not being used were conditionally compiled out. The capability of supporting children did not appear to be needed and was not implemented in the rtems version of these routines. == NICs Status == ---- Driver Symbol Status ====== ====== ====== RealTek _bsd_re_pcimodule_sys_init Links EtherExpress _bsd_fxp_pcimodule_sys_init Links DEC tulip _bsd_dc_pcimodule_sys_init Links Broadcom BCM57xxx _bsd_bce_pcimodule_sys_init Links Broadcom BCM4401 _bsd_bfe_pcimodule_sys_init Links Broadcom BCM570x _bsd_bge_pcimodule_sys_init Needs Symbols (A) E1000 IGB _bsd_igb_pcimodule_sys_init Links E1000 EM _bsd_em_pcimodule_sys_init Links Cadence ? Links, works. ---- To add a NIC edit rtemsbsd/include/bsp/nexus-devices.h and add the driver reference to the architecture and/or BSP. For example to add the RealTek driver add: SYSINIT_DRIVER_REFERENCE(re, pci); and to add the MII PHY driver add: SYSINIT_DRIVER_REFERENCE(rge, miibus); The PC BSP has these entries. Symbols (A) pci_get_vpd_ident === Cadence === The cadence driver works on the Xilinx Zynq platform. The hardware checksum support works on real hardware but does not seem to be supported on qemu therefore the default state is to disable TXCSUM and RXCSUM and this can be enabled from the shell with: # ifconfig cgem0 rxcsum txcsum or with an ioctl call to the network interface driver with SIOCSIFCAP and the mask IFCAP_TXCSUM and IFCAP_RXCSUM set. == PF (Firewall) == It is possible to use PF as a firewall. See [https://www.freebsd.org/doc/handbook/firewalls-pf.html] for details on the range of functions and for how to configure the firewall. The following is necessary to use PF on RTEMS: - You have to provide a +/etc/pf.os+ file. The firewall can use it for passive OS fingerprinting. If you don't want to use this feature, the file may contain nothing except a line of comment (for example "# empty"). - If some filters use protocol names (like tcp or udp) you have to provide a +/etc/protocols+ file. - If some filters use service names (like ssh or http) you have to provide a +/etc/services+ file. - Create a rule file (normally +/etc/pf.conf+). See the FreeBSD manual for the syntax. - Load the rule file using the pfctl command and enable pf. Please note that the pfctl command needs a lot of stack. You should use at least RTEMS_MINIMUM_STACK_SIZE + 8192 Bytes of stack. An example initialisation can look like follows: ---- int exit_code; char *params[] = { "pfctl", "-f", "/etc/pf.conf", "-e", NULL }; exit_code = rtems_bsd_command_pfctl(ARGC(params), params); assert(exit_code == EXIT_SUCCSESS); ---- === Known restrictions === - Currently PF on RTEMS always uses the configuration for memory restricted systems (on FreeBSD that means systems with less than 100 MB RAM). This is fixed in +pfctl_init_options()+. == Problems to report to FreeBSD == The MMAP_NOT_AVAILABLE define is inverted on its usage. When it is defined the mmap method is called. Additionally, it is not used thoroughly. It is not used in the unmap portion of the source. The file rec_open.c uses the define MMAP_NOT_AVAILABLE to wrap the call to mmap and file rec_close.c uses the munmap method.