source: rtems-libbsd/libbsd.txt @ 28c973a

55-freebsd-126-freebsd-12freebsd-9.3
Last change on this file since 28c973a was 28c973a, checked in by Sebastian Huber <sebastian.huber@…>, on 05/06/16 at 07:31:20

Add "How to import code from FreeBSD"

  • Property mode set to 100644
File size: 41.4 KB
Line 
1RTEMS BSD Library Guide
2=======================
3:toc:
4:icons:
5:numbered:
6:website: http://www.rtems.org/
7
8RTEMS uses FreeBSD 9.2 as the source of its TCP/IP and USB stacks.
9This is a guide which captures information on the
10process of merging code from FreeBSD, building this library,
11RTEMS specific support files, and general guidelines on what
12modifications to the FreeBSD source are permitted.
13
14Goals of this effort are
15
16* update TCP/IP and provide USB in RTEMS,
17* ease updating to future FreeBSD versions,
18* ease tracking changes in FreeBSD code,
19* minimize manual changes in FreeBSD code, and
20* define stable kernel/device driver API which is implemented
21by both RTEMS and FreeBSD. This is the foundation of the port.
22
23We will work to push our changes upstream to the FreeBSD Project
24and minimize changes required at each update point.
25
26*******************************************************************************
27This is a work in progress and is very likely to be incomplete.
28Please help by adding to it.
29*******************************************************************************
30
31== Getting Started
32
33=== Tool Chain ===
34
35You need a tool chain for RTEMS based on at least RSB 4.12 April 2016 or later.
36=== Installation Overview ===
37
38. You must configure your BSP with the +--disable-networking+ option to disable
39the old network stack.  Make sure no header files of the old network stack are
40installed.
41
42. Clone the Git repository +git clone git://git.rtems.org/rtems-libbsd.git+.
43. Change into the RTEMS BSD library root directory.
44. Edit the `config.inc` configuration file and adjust it to your environment.
45. Run +waf configure ...+.
46. Run +waf+.
47. Run +waf install+.
48
49Refer to the README.waf for Waf building instructions.
50
51Make sure the submodules have been initialised and are updated. If a 'git
52status' says `rtems_waf` need updating run the submodule update command:
53
54 $ git submodule rtems_waf update
55
56=== Board Support Package Requirements ===
57
58The RTEMS version must be at least 4.11.  The Board Support Package (BSP)
59should support the
60http://www.rtems.org/onlinedocs/doxygen/cpukit/html/group\__rtems\__interrupt__extension.html[Interrupt Manager Extension]
61// The first underscores have to be masked to stop asciidoc interpreting them
62to make use of generic FreeBSD based drivers.
63
64The linker command file of the BSP must contain the following section
65definitions:
66
67-------------------------------------------------------------------------------
68.rtemsroset : {
69        KEEP (*(SORT(.rtemsroset.*)))
70}
71
72.rtemsrwset : {
73        KEEP (*(SORT(.rtemsrwset.*)))
74}
75-------------------------------------------------------------------------------
76
77The first output section can be placed in read-only memory.  The second output
78section must be placed in read-write memory.  The output section name is not
79relevant.  The output sections may also contain other input sections.
80
81=== Board Support Package Configuration and Build ===
82
83You need to configure RTEMS for the desired BSP and install it.  The BSP should
84be configured with a disabled network stack.  The BSD library containing the
85new network stack is a separate package.  Using a BSP installation containing
86the old network stack may lead to confusion and unpredictable results.
87
88The following script is used to build the `arm/realview_pbx_a9_qemu` BSP for
89our internal testing purposes:
90
91-------------------------------------------------------------------------------
92#!/bin/sh
93
94cd ${HOME}/sandbox
95rm -rf b-realview_pbx_a9_qemu
96mkdir b-realview_pbx_a9_qemu
97cd b-realview_pbx_a9_qemu
98${HOME}/git-rtems/configure \
99        --prefix=${HOME}/sandbox/install \
100        --target=arm-rtems4.11 \
101        --enable-rtemsbsp=realview_pbx_a9_qemu \
102        --disable-networking && \
103        make && \
104        make install
105-------------------------------------------------------------------------------
106
107The `arm/realview_pbx_a9_qemu` BSP running on the Qemu simulator has some
108benefits for development and test of the BSD library
109
110* it offers a NULL pointer read and write protection,
111* Qemu is a fast simulator,
112* Qemu provides support for GDB watchpoints,
113* Qemu provides support for virtual Ethernet networks, e.g. TUN and bridge
114devices (you can run multiple test instances on one virtual network).
115
116=== BSD Library Configuration and Build ===
117
118The build system based on the Waf build system. To build with Waf please refer
119to the README.waf file.
120
121===== Example Configuration =====
122
123In the BSD library source directory edit the file `config.inc`.  Continuing on
124the above, the `config.inc` used to match the above is:
125
126-------------------------------------------------------------------------------
127# Mandatory: Select your BSP and installation prefix
128TARGET = arm-rtems4.11
129BSP = realview_pbx_a9_qemu
130PREFIX = $(HOME)/sandbox/install
131
132# Optional: Separate installation base directory
133INSTALL_BASE = $(PREFIX)/$(TARGET)/$(BSP)
134
135# Optional: Network test configuration
136TEST_RUNNER = $(BSP)
137NET_CFG_SELF_IP = 10.0.0.2
138NET_CFG_NETMASK = 255.255.0.0
139NET_CFG_PEER_IP = 10.0.0.1
140NET_CFG_GATEWAY_IP = 10.0.0.1
141NET_TAP_INTERFACE = tap0
142-------------------------------------------------------------------------------
143
144=== BSD Library Initialization ===
145
146Use the following code to initialize the BSD library:
147
148-------------------------------------------------------------------------------
149#include <assert.h>
150#include <sysexits.h>
151
152#include <machine/rtems-bsd-commands.h>
153#include <rtems/bsd/bsd.h>
154
155static void
156network_ifconfig_lo0(void)
157{
158        int exit_code;
159        char *lo0[] = {
160                "ifconfig",
161                "lo0",
162                "inet",
163                "127.0.0.1",
164                "netmask",
165                "255.255.255.0",
166                NULL
167        };
168        char *lo0_inet6[] = {
169                "ifconfig",
170                "lo0",
171                "inet6",
172                "::1",
173                "prefixlen",
174                "128",
175                NULL
176        };
177
178        exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0), lo0);
179        assert(exit_code == EX_OK);
180
181        exit_code = rtems_bsd_command_ifconfig(RTEMS_BSD_ARGC(lo0_inet6), lo0_inet6);
182        assert(exit_code == EX_OK);
183}
184
185void
186network_init(void)
187{
188        rtems_status_code sc;
189
190        sc = rtems_bsd_initialize();
191        assert(sc == RTEMS_SUCCESSFUL);
192
193        network_ifconfig_lo0();
194}
195-------------------------------------------------------------------------------
196
197This performs the basic network stack initialization with a loopback interface.
198Further initialization must be done using the standard BSD network
199configuration commands
200http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)]
201using `rtems_bsd_command_ifconfig()` and
202http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)]
203using `rtems_bsd_command_route()`.  For an example please have a look at
204`testsuite/include/rtems/bsd/test/default-network-init.h`.
205
206=== Task Priorities and Stack Size ===
207
208The default task priority is 96 for the interrupt server task (name "IRQS"), 98
209for the timer server task (name "TIME") and 100 for all other tasks.  The
210application may provide their own implementation of the
211`rtems_bsd_get_task_priority()` function (for example in the module which calls
212`rtems_bsd_initialize()`) if different values are desired.
213
214The task stack size is determined by the `rtems_bsd_get_task_stack_size()`
215function which may be provided by the application in case the default is not
216appropriate.
217
218=== Size for Allocator Domains ===
219
220The size for an allocator domain can be specified via the
221`rtems_bsd_get_allocator_domain_size()` function.  The application may provide
222their own implementation of the `rtems_bsd_get_allocator_domain_size()`
223function (for example in the module which calls `rtems_bsd_initialize()`) if
224different values are desired.  The default size is 8MiB for all domains.
225
226== Network Stack Features
227
228http://roy.marples.name/projects/dhcpcd/index[DHCPCD(8)]:: DHCP client
229
230https://developer.apple.com/library/mac/documentation/Networking/Reference/DNSServiceDiscovery_CRef/Reference/reference.html[dns_sd.h]:: DNS Service Discovery
231
232http://www.opensource.apple.com/source/mDNSResponder/mDNSResponder-320.10/mDNSCore/mDNSEmbeddedAPI.h[mDNS]:: Multi-Cast DNS
233
234http://www.freebsd.org/cgi/man.cgi?query=unix&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[UNIX(4)]:: UNIX-domain protocol family
235
236http://www.freebsd.org/cgi/man.cgi?query=inet&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[INET(4)]:: Internet protocol family
237
238http://www.freebsd.org/cgi/man.cgi?query=inet6&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[INET6(4)]:: Internet protocol version 6 family
239
240http://www.freebsd.org/cgi/man.cgi?query=tcp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[TCP(4)]:: Internet Transmission Control Protocol
241
242http://www.freebsd.org/cgi/man.cgi?query=udp&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[UDP(4)]:: Internet User Datagram Protocol
243
244http://www.freebsd.org/cgi/man.cgi?query=route&sektion=4&apropos=0&manpath=FreeBSD+9.2-RELEASE[ROUTE(4)]:: Kernel packet forwarding database
245
246http://www.freebsd.org/cgi/man.cgi?query=bpf&apropos=0&sektion=4&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[BPF(4)]:: Berkeley Packet Filter
247
248http://www.freebsd.org/cgi/man.cgi?query=socket&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SOCKET(2)]:: Create an endpoint for communication
249
250http://www.freebsd.org/cgi/man.cgi?query=kqueue&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[KQUEUE(2)]:: Kernel event notification mechanism
251
252http://www.freebsd.org/cgi/man.cgi?query=select&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[SELECT(2)]:: Synchronous I/O multiplexing
253
254http://www.freebsd.org/cgi/man.cgi?query=poll&apropos=0&sektion=2&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[POLL(2)]:: Synchronous I/O multiplexing
255
256http://www.freebsd.org/cgi/man.cgi?query=route&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[ROUTE(8)]:: Manually manipulate the routing tables
257
258http://www.freebsd.org/cgi/man.cgi?query=ifconfig&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[IFCONFIG(8)]:: Configure network interface parameters
259
260http://www.freebsd.org/cgi/man.cgi?query=netstat&apropos=0&sektion=1&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[NETSTAT(1)]:: Show network status
261
262http://www.freebsd.org/cgi/man.cgi?query=ping&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING(8)]:: Send ICMP ECHO_REQUEST packets to network hosts
263
264http://www.freebsd.org/cgi/man.cgi?query=ping6&apropos=0&sektion=8&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[PING6(8)]:: Send ICMPv6 ECHO_REQUEST packets to network hosts
265
266http://www.freebsd.org/cgi/man.cgi?query=sysctl&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[SYSCTL(3)]:: Get or set system information
267
268http://www.freebsd.org/cgi/man.cgi?query=resolver&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[RESOLVER(3)]:: Resolver routines
269
270http://www.freebsd.org/cgi/man.cgi?query=gethostbyname&sektion=3&apropos=0&manpath=FreeBSD+9.2-RELEASE[GETHOSTBYNAME(3)]:: Get network host entry
271
272== Network Interface Drivers
273
274=== Link Up/Down Events
275
276You can notifiy the application space of link up/down events in your network
277interface driver via the if_link_state_change(LINK_STATE_UP/LINK_STATE_DOWN)
278function.  The DHCPCD(8) client is a consumer of these events for example.
279Make sure that the interface flag IFF_UP and the interface driver flag
280IFF_DRV_RUNNING is set in case the link is up, otherwise ether_output() will
281return the error status ENETDOWN.
282
283== Shell Commands
284
285=== HOSTNAME(1)
286
287In addition to the standard options the RTEMS version of the HOSTNAME(1)
288command supports the -m flag to set/get the multicast hostname of the
289mDNS resolver instance.  See also rtems_mdns_sethostname() and
290rtems_mdns_gethostname().
291
292== Qemu
293
294Use the following script to set up a virtual network with three tap devices
295connected via one bridge device.
296
297-------------------------------------------------------------------------------
298#!/bin/sh -x
299
300user=`whoami`
301interfaces=(1 2 3)
302
303tap=qtap
304bri=qbri
305
306case $1 in
307        up)
308                sudo -i brctl addbr $bri
309                for i in ${interfaces[@]} ; do
310                        sudo -i tunctl -t $tap$i -u $user ;
311                        sudo -i ifconfig $tap$i up ;
312                        sudo -i brctl addif $bri $tap$i ;
313                done
314                sudo -i ifconfig $bri up
315                ;;
316        down)
317                for i in ${interfaces[@]} ; do
318                        sudo -i ifconfig $tap$i down ;
319                        sudo -i tunctl -d $tap$i ;
320                done
321                sudo -i ifconfig $bri down
322                sudo -i brctl delbr $bri
323                ;;
324esac
325-------------------------------------------------------------------------------
326
327Connect your Qemu instance to one of the tap devices, e.g.
328
329-------------------------------------------------------------------------------
330qemu-system-i386 -m 512 -boot a -cpu pentium3 \
331        -drive file=$HOME/qemu/pc386_fda,index=0,if=floppy,format=raw \
332        -drive file=fat:$HOME/qemu/hd,format=raw \
333        -net nic,model=e1000,macaddr=0e:b0:ba:5e:ba:11 \
334        -net tap,ifname=qtap1,script=no,downscript=no \
335        -nodefaults -nographic -serial stdio
336-------------------------------------------------------------------------------
337
338Make sure that each Qemu instance uses its own MAC address to avoid an address
339conflict (or otherwise use it as a test).
340
341To connect the Qemu instances with your local network use the following
342(replace 'eth0' with the network interface of your host).
343
344-------------------------------------------------------------------------------
345ifconfig eth0 0.0.0.0
346brctl addif qbri eth0
347dhclient qbri
348-------------------------------------------------------------------------------
349
350== Issues and TODO
351
352* PCI support on x86 uses a quick and dirty hack, see pci_reserve_map().
353
354* Priority queues are broken with clustered scheduling.
355
356* Per-CPU data should be enabled once the new stack is ready for SMP.
357
358* Per-CPU NETISR(9) should be enabled onece the new stack is ready for SMP.
359
360* Multiple routing tables are not supported.  Every FIB value is set to zero
361  (= BSD_DEFAULT_FIB).
362
363* Process identifiers are not supported.  Every PID value is set to zero
364  (= BSD_DEFAULT_PID).
365
366* User credentials are not supported.  The following functions allow the
367  operation for everyone
368  - prison_equal_ip4(),
369  - chgsbsize(),
370  - cr_cansee(),
371  - cr_canseesocket() and
372  - cr_canseeinpcb().
373
374* A basic USB functionality test that is known to work on Qemu is desirable.
375
376* Adapt generic IRQ PIC interface code to Simple Vectored Interrupt Model
377  so that those architectures can use new TCP/IP and USB code.
378
379* freebsd-userspace/rtems/include/sys/syslog.h is a copy from the old
380  RTEMS TCP/IP stack. For some reason, the __printflike markers do not
381  compile in this environment. We may want to use the FreeBSD syslog.h
382  and get this addressed.
383
384* in_cksum implementations for architectures not supported by FreeBSD.
385  This will require figuring out where to put implementations that do
386  not originate from FreeBSD and are populated via the script.
387
388* MAC support functions are not thread-safe ("freebsd/lib/libc/posix1e/mac.c").
389
390* IFCONFIG(8): IEEE80211 support is disabled.  This module depends on a XML
391  parser and mmap().
392
393* get_cyclecount(): The implementation is a security problem.
394
395* What to do with the priority parameter present in the FreeBSD synchronization
396  primitives and the thread creation functions?
397
398* TASKQUEUE(9): Support spin mutexes.
399
400* ZONE(9): Review allocator lock usage in rtems-bsd-chunk.c.
401
402* KQUEUE(2): Choose proper lock for global kqueue list.
403
404* TIMEOUT(9): Maybe use special task instead of timer server to call
405  callout_tick().
406
407* sysctl_handle_opaque(): Implement reliable snapshots.
408
409* PING6(8): What to do with SIGALARM?
410
411* <sys/param.h>: Update Newlib to use a MSIZE of 256.
412
413* BPF(4): Add support for zero-copy buffers.
414
415* UNIX(4): Fix race conditions in the area of socket object and file node
416  destruction.  Add support for file descriptor transmission via control
417  messages.
418
419* PRINTF(9): Add support for log(), the %D format specifier is missing in the
420  normal printf() family.
421
422* Why is the interrupt server used?  The BSD interrupt handlers can block on
423synchronization primitives like mutexes.  This is in contrast to RTEMS
424interrupt service routines.  The BSPs using the generic interrupt support must
425implement the `bsp_interrupt_vector_enable()` and
426`bsp_interrupt_vector_disable()` routines.  They normally enable/disable a
427particular interrupt source at the interrupt controller.  This can be used to
428implement the interrupt server.  The interrupt server is a task that wakes-up
429in case an associated interrupt happens.  The interrupt source is disabled in
430a generic interrupt handler that wakes-up the interrupt server task.   Once the
431postponed interrupt processing is performed in the interrupt server the
432interrupt source is enabled again.
433
434* Convert all BSP linkcmds to use a linkcmds.base so the sections are
435easier to insert.
436
437* NIC Device Drivers
438- Only common PCI NIC drivers have been included in the initial set. These
439do not include any system on chip or ISA drivers.
440- PCI configuration probe does not appear to happen to determine if a
441NIC is in I/O or memory space. We have worked around this by using a
442static hint to tell the fxp driver the correct mode. But this needs to
443be addressed.
444- The ISA drivers require more BSD infrastructure to be addressed. This was
445outside the scope of the initial porting effort.
446
447== FreeBSD Source
448
449You should be able to rely on FreebSD manual pages and documentation
450for details on the code itself.
451
452=== Automatically Generated FreeBSD Files
453
454Some source and header files are automatically generated during the FreeBSD
455build process.  The `Makefile.todo` file performs this manually.  The should be
456included in `freebsd-to-rtems.py` script some time in the future.  For details,
457see also
458http://www.freebsd.org/cgi/man.cgi?query=kobj&sektion=9&apropos=0&manpath=FreeBSD+9.2-RELEASE[KOBJ(9)].
459
460=== Rules for Modifying FreeBSD Source
461
462Only add lines.  If your patch contains lines starting with a '-', then this is
463wrong.  Subtract code by added `#ifndef __rtems__`.  This makes merging easier
464in the future.  For example:
465
466-------------------------------------------------------------------------------
467/* Global variables for the kernel. */
468
469#ifndef __rtems__
470/* 1.1 */
471extern char kernelname[MAXPATHLEN];
472#endif /* __rtems__ */
473
474extern int tick;                        /* usec per tick (1000000 / hz) */
475-------------------------------------------------------------------------------
476
477-------------------------------------------------------------------------------
478#if defined(_KERNEL) || defined(_WANT_FILE)
479#ifdef __rtems__
480#include <rtems/libio_.h>
481#include <sys/fcntl.h>
482#endif /* __rtems__ */
483/*
484 * Kernel descriptor table.
485 * One entry for each open kernel vnode and socket.
486 *
487 * Below is the list of locks that protects members in struct file.
488 *
489 * (f) protected with mtx_lock(mtx_pool_find(fp))
490 * (d) cdevpriv_mtx
491 * none not locked
492 */
493-------------------------------------------------------------------------------
494
495-------------------------------------------------------------------------------
496extern int profprocs;                   /* number of process's profiling */
497#ifndef __rtems__
498extern volatile int ticks;
499#else /* __rtems__ */
500#include <rtems/score/watchdogimpl.h>
501#define ticks _Watchdog_Ticks_since_boot
502#endif /* __rtems__ */
503
504#endif /* _KERNEL */
505-------------------------------------------------------------------------------
506
507Add nothing (even blank lines) before or after the `__rtems__` guards.  Always
508include a `__rtems__` in the guards to make searches easy, so use
509
510* `#ifndef __rtems__`,
511* `#ifdef __rtems__`,
512* `#else /* __rtems__ */`, and
513* `#endif /* __rtems__ */`.
514
515For new code use
516http://www.freebsd.org/cgi/man.cgi?query=style&apropos=0&sektion=9&manpath=FreeBSD+9.2-RELEASE&arch=default&format=html[STYLE(9)].
517Do not format original FreeBSD code.
518
519== BSD Library Source
520
521=== What is in the Git Repository
522
523There is a self-contained kit with FreeBSD and RTEMS components pre-merged. The
524Waf wscript in this kit is automatically generated.
525
526Any changes to source in the `freebsd` directories will need to be merged
527upstream into our master FreeBSD checkout, the `freebsd-org` submodule.
528
529The repository contains two FreeBSD source trees.  In the `freebsd` directory
530are the so called 'managed' FreeBSD sources used to build the BSD library.  The
531FreeBSD source in `freebsd-org` is the 'master' version.  The
532`freebsd-to-rtems.py` script is used to transfer files between the two trees.
533In general terms, if you have modified managed FreeBSD sources, you will need
534to run the script in 'revert' or 'reverse' mode using the `-R` switch.  This
535will copy the source back to your local copy of the master FreeBSD source so
536you can run `git diff` against the upstream FreeBSD source.  If you want to
537transfer source files from the master FreeBSD source to the manged FreeBSD
538sources, then you must run the script in 'forward' mode (the default).
539
540=== Organization
541
542The top level directory contains a few directories and files. The following
543are important to understand
544
545* `freebsd-to-rtems.py` - script to convert to and free FreeBSD and RTEMS trees,
546* `create-kernel-namespace.sh` - script to create the kernel namespace header <machine/rtems-bsd-kernel-namespace.h,
547* `wscript` - automatically generated,
548* `freebsd/` - from FreeBSD by script,
549* `rtemsbsd/` - RTEMS specific implementations of FreeBSD kernel support routines,
550* `testsuite/` - RTEMS specific tests, and
551* `libbsd.txt` - documentation in Asciidoc.
552
553== Moving Code Between Managed and Master FreeBSD Source
554
555The script `freebsd-to-rtems.py` is used to copy code from FreeBSD to the
556rtems-libbsd tree and to reverse this process. This script attempts to
557automate this process as much as possible and performs some transformations
558on the FreeBSD code. Its command line arguments are shown below:
559
560----
561freebsd-to-rtems.py [args]
562  -?|-h|--help      print this and exit
563  -d|--dry-run      run program but no modifications
564  -D|--diff         provide diff of files between trees
565  -e|--early-exit   evaluate arguments, print results, and exit
566  -m|--makefile     Warning: depreciated and will be removed
567  -b|--buildscripts just generate the build scripts
568  -R|--reverse      default FreeBSD -> RTEMS, reverse that
569  -r|--rtems        RTEMS directory
570  -f|--freebsd      FreeBSD directory
571  -v|--verbose      enable verbose output mode
572----
573
574In its default mode of operation, freebsd-to-rtems.py is used to copy code
575from FreeBSD to the rtems-libbsd tree and perform transformations.  In forward
576mode, the script may be requested to just generate the Waf script.
577
578In "reverse mode", this script undoes those transformations and copies
579the source code back to the FreeBSD SVN tree. This allows us to do
580'svn diff', evaluate changes made by the RTEMS Project, and report changes
581back to FreeBSD upstream.
582
583In either mode, the script may be asked to perform a dry-run or be verbose.
584Also, in either mode, the script is also smart enough to avoid copying over
585files which have not changed. This means that the timestamps of files are
586not changed unless the contents change. The script will also report the
587number of files which changed. In verbose mode, the script will print
588the name of the files which are changed.
589
590To add or update files int the RTEMS FreeBSD tree first run the 'reverse mode'
591and move the current set of patches FreeBSD. The script may warn you if a file
592is not present at the destination for the direction. This can happen as files
593not avaliable at the FreeBSD snapshot point have been specially added to the
594RTEMS FreeBSD tree. Warnings can also appear if you have changed the list of
595files in libbsd.py. The reverse mode will result in the FreeBSD having
596uncommitted changes. You can ignore these. Once the reverse process has
597finished edit libbsd.py and add any new files then run the forwad mode to bring
598those files into the RTEMS FreeBSD tree.
599
600The following is an example forward run with no changes.
601
602----
603$ ~/newbsd/git/libbsd-8.2/freebsd-to-rtems.py \
604    -r /home/joel/newbsd/git/libbsd-8.2 \
605    -f /home/joel/newbsd/libbsd/freebsd-8.2 -v
606Verbose:                yes (1)
607Dry Run:                no
608Only Generate Makefile: no
609RTEMS Directory:        /home/joel/newbsd/git/libbsd-8.2
610FreeBSD Directory:      /home/joel/newbsd/libbsd/freebsd-8.2
611Direction:              forward
612Generating into /home/joel/newbsd/git/libbsd-8.2
6130 files were changed.
614----
615
616The script may also be used to generate a diff in either forward or reverse
617direction.
618
619You can add more than one verbose option (-v) to the command line and get more
620detail and debug level information from the command.
621
622== How to import code from FreeBSD
623
624. Run `git status` and make sure your working directory is clean.
625. Run `./freebsd-to-rtems.py -R`
626. Run `./freebsd-to-rtems.py`
627. Run `git status` and make sure your working directory is clean.  If you see modified files, then the `freebsd-to-rtems.py` script needs to be fixed first.
628. Add the files to import to `libbsd.py`.
629. Run `./freebsd-to-rtems.py`
630. Immediately check in the imported files without the changes to `libbsd_waf.py`.  Do not touch the imported files yourself at this point.
631. Port the imported files to RTEMS.  See 'Rules for Modifying FreeBSD Source'.
632. Add a test to the testsuite if possible.
633. Run `./create-kernel-namespace.sh` if you imported kernel space headers.  Add only your new defines via `git add -p rtemsbsd/include/machine/rtems-bsd-kernel-namespace.h`.
634. Create one commit from this.
635
636== Initialization of the BSD Library
637
638The initialization of the BSD library is based on the FreeBSD SYSINIT(9)
639infrastructure.  The key to initializing a system is to ensure that the desired
640device drivers are explicitly pulled into the linked application.  This plus
641linking against the BSD library (`libbsd.a`) will pull in the necessary FreeBSD
642infrastructure.
643
644The FreeBSD kernel is not a library like the RTEMS kernel.  It is a bunch of
645object files linked together.  If we have a library, then creating the
646executable is simple.  We begin with a start symbol and recursively resolve all
647references.  With a bunch of object files linked together we need a different
648mechanism.  Most object files don't know each other.  Lets say we have a driver
649module.  The rest of the system has no references to this driver module.  The
650driver module needs a way to tell the rest of the system: Hey, kernel I am
651here, please use my services!
652
653This registration of independent components is performed by SYSINIT(9) and
654specializations:
655
656http://www.freebsd.org/cgi/man.cgi?query=SYSINIT
657
658The SYSINIT(9) uses some global data structures that are placed in a certain
659section.  In the linker command file we need this:
660
661-------------------------------------------------------------------------------
662.rtemsroset : {
663        KEEP (*(SORT(.rtemsroset.*)))
664}
665
666.rtemsrwset : {
667        KEEP (*(SORT(.rtemsrwset.*)))
668}
669-------------------------------------------------------------------------------
670
671This results for example in this executable layout:
672
673-------------------------------------------------------------------------------
674[...]
675 *(SORT(.rtemsroset.*))
676 .rtemsroset.bsd.modmetadata_set.begin
677                0x000000000025fe00        0x0 libbsd.a(rtems-bsd-init.o)
678                0x000000000025fe00                _bsd__start_set_modmetadata_set
679 .rtemsroset.bsd.modmetadata_set.content
680                0x000000000025fe00        0x8 libbsd.a(rtems-bsd-nexus.o)
681 .rtemsroset.bsd.modmetadata_set.content
682                0x000000000025fe08        0x4 libbsd.a(kern_module.o)
683[...]
684 .rtemsroset.bsd.modmetadata_set.content
685                0x000000000025fe68        0x4 libbsd.a(mii.o)
686 .rtemsroset.bsd.modmetadata_set.content
687                0x000000000025fe6c        0x4 libbsd.a(mii_bitbang.o)
688 .rtemsroset.bsd.modmetadata_set.end
689                0x000000000025fe70        0x0 libbsd.a(rtems-bsd-init.o)
690                0x000000000025fe70                _bsd__stop_set_modmetadata_set
691[...]
692.rtemsrwset     0x000000000030bad0      0x290
693 *(SORT(.rtemsrwset.*))
694 .rtemsrwset.bsd.sysinit_set.begin
695                0x000000000030bad0        0x0 libbsd.a(rtems-bsd-init.o)
696                0x000000000030bad0                _bsd__start_set_sysinit_set
697 .rtemsrwset.bsd.sysinit_set.content
698                0x000000000030bad0        0x4 libbsd.a(rtems-bsd-nexus.o)
699 .rtemsrwset.bsd.sysinit_set.content
700                0x000000000030bad4        0x8 libbsd.a(rtems-bsd-thread.o)
701 .rtemsrwset.bsd.sysinit_set.content
702                0x000000000030badc        0x4 libbsd.a(init_main.o)
703[...]
704 .rtemsrwset.bsd.sysinit_set.content
705                0x000000000030bd54        0x4 libbsd.a(frag6.o)
706 .rtemsrwset.bsd.sysinit_set.content
707                0x000000000030bd58        0x8 libbsd.a(uipc_accf.o)
708 .rtemsrwset.bsd.sysinit_set.end
709                0x000000000030bd60        0x0 libbsd.a(rtems-bsd-init.o)
710                0x000000000030bd60                _bsd__stop_set_sysinit_set
711[...]
712-------------------------------------------------------------------------------
713
714Here you can see, that some global data structures are collected into
715continuous memory areas.  This memory area can be identified by start and stop
716symbols.  This constructs a table of uniform items.
717
718The low level FreeBSD code calls at some time during the initialization the
719mi_startup() function (machine independent startup).  This function will sort
720the SYSINIT(9) set and call handler functions which perform further
721initialization.  The last step is the scheduler invocation.
722
723The SYSINIT(9) routines are run in mi_startup() which is called by
724rtems_bsd_initialize().
725
726This is also explained in "The Design and Implementation of the FreeBSD
727Operating System" section 14.3 "Kernel Initialization".
728
729In RTEMS we have a library and not a bunch of object files.  Thus we need a way
730to pull-in the desired services out of the libbsd.  Here the
731`rtems-bsd-sysinit.h` comes into play.  The SYSINIT(9) macros have been
732modified and extended for RTEMS in `<sys/kernel.h>`:
733
734-------------------------------------------------------------------------------
735#ifndef __rtems__
736#define C_SYSINIT(uniquifier, subsystem, order, func, ident)    \
737        static struct sysinit uniquifier ## _sys_init = {       \
738                subsystem,                                      \
739                order,                                          \
740                func,                                           \
741                (ident)                                         \
742        };                                                      \
743        DATA_SET(sysinit_set,uniquifier ## _sys_init)
744#else /* __rtems__ */
745#define SYSINIT_ENTRY_NAME(uniquifier)                          \
746        _bsd_ ## uniquifier ## _sys_init
747#define SYSINIT_REFERENCE_NAME(uniquifier)                      \
748        _bsd_ ## uniquifier ## _sys_init_ref
749#define C_SYSINIT(uniquifier, subsystem, order, func, ident)    \
750        struct sysinit SYSINIT_ENTRY_NAME(uniquifier) = {       \
751                subsystem,                                      \
752                order,                                          \
753                func,                                           \
754                (ident)                                         \
755        };                                                      \
756        RWDATA_SET(sysinit_set,SYSINIT_ENTRY_NAME(uniquifier))
757#define SYSINIT_REFERENCE(uniquifier)                           \
758        extern struct sysinit SYSINIT_ENTRY_NAME(uniquifier);   \
759        static struct sysinit const * const                     \
760        SYSINIT_REFERENCE_NAME(uniquifier) __used               \
761        = &SYSINIT_ENTRY_NAME(uniquifier)
762#define SYSINIT_MODULE_REFERENCE(mod)                           \
763        SYSINIT_REFERENCE(mod ## module)
764#define SYSINIT_DRIVER_REFERENCE(driver, bus)                   \
765        SYSINIT_MODULE_REFERENCE(driver ## _ ## bus)
766#define SYSINIT_DOMAIN_REFERENCE(dom)                           \
767        SYSINIT_REFERENCE(domain_add_ ## dom)
768#endif /* __rtems__ */
769-------------------------------------------------------------------------------
770
771Here you see that the SYSINIT(9) entries are no longer static.  The
772\*_REFERENCE() macros will create references to the corresponding modules which
773are later resolved by the linker.  The application has to provide an object
774file with references to all required FreeBSD modules.
775
776The FreeBSD device model is quite elaborated (with follow-ups):
777
778http://www.freebsd.org/cgi/man.cgi?query=driver
779
780The devices form a tree with the Nexus device at a high-level.  This Nexus
781device is architecture specific in FreeBSD.  In RTEMS we have our own Nexus
782device, see `rtemsbsd/bsp/bsp-bsd-nexus-devices.c`.
783
784=== SYSCTL_NODE Example
785
786During development, we had an undefined reference to
787_bsd_sysctl__net_children that we had trouble tracking down. Thanks to
788Chris Johns, we located it. He explained how to read SYSCTL_NODE
789definitions. This line from freebsd/netinet/in_proto.c is attempting
790to add the "inet" node to the parent node "_net".
791
792----
793SYSCTL_NODE(_net,      PF_INET,         inet,   CTLFLAG_RW, 0,
794        "Internet Family");
795----
796
797Our problem was that we could not find where _bsd_sysctl__net_children
798was defined. Chris suggested that when in doubt compile with -save-temps
799and look at the preprocessed .i files. But he did not need that. He
800explained that this the symbol name _bsd_sysctl__net_children was
801automatically generated by a SYSCTL_NODE as follows:
802
803* _bsd_ - added by RTEMS modifications to SYSCTL_NODE macro
804* sysctl_ - boilerplace added by SYSCTL_NODE macro
805* "" - empty string for parent node
806* net - name of SYSCTL_NODE
807* children - added by SYSCTL macros
808
809This was all generated by a support macro declaring the node as this:
810
811----
812struct sysctl_oid_list SYSCTL_NODE_CHILDREN(parent, name);
813----
814
815Given this information, we located this SYSCTL_NODE declaration in
816kern/kern_mib.c
817
818----
819SYSCTL_NODE(, CTL_KERN,   kern,   CTLFLAG_RW, 0,
820        "High kernel, proc, limits &c");
821----
822
823== Core FreeBSD APIs and RTEMS Replacements ==
824
825=== SX(9) (Shared/exclusive locks) ===
826
827http://www.freebsd.org/cgi/man.cgi?query=sx
828
829Binary semaphores (this neglects the ability to allow shared access).
830
831=== MUTEX(9) (Mutual exclusion) ===
832
833http://www.freebsd.org/cgi/man.cgi?query=mutex
834
835Binary semaphores (not recursive mutexes are not supported this way).
836
837=== RWLOCK(9) (Reader/writer lock) ===
838
839http://www.freebsd.org/cgi/man.cgi?query=rwlock
840
841POSIX r/w lock.
842
843=== RMLOCK(9) (Reader/writer lock optimized for mostly read access patterns) ===
844
845Note:  This object was implemented as a wrapper for RWLOCK in the rm_lock header file.
846
847http://www.freebsd.org/cgi/man.cgi?query=rmlock
848
849POSIX r/w lock.
850
851=== CONDVAR(9) (Condition variables) ===
852
853http://www.freebsd.org/cgi/man.cgi?query=condvar
854
855POSIX condition variables with modifications (hack).
856
857=== CALLOUT(9) (Timer functions) ===
858
859http://www.freebsd.org/cgi/man.cgi?query=callout
860
861Timer server.
862
863=== TASKQUEUE(9) (Asynchronous task execution) ===
864
865http://www.freebsd.org/cgi/man.cgi?query=taskqueue
866
867TBD.
868
869=== KTHREAD(9), KPROC(9) (Tasks) ===
870
871http://www.freebsd.org/cgi/man.cgi?query=kthread
872
873http://www.freebsd.org/cgi/man.cgi?query=kproc
874
875Tasks.
876
877=== ZONE(9) (Zone allocator) ===
878
879http://www.freebsd.org/cgi/man.cgi?query=zone
880
881TBD.
882
883=== devfs (Device file system) ===
884
885Dummy, IMFS or new implementation (currently dummy).
886
887=== psignal (Signals) ===
888
889TBD.  Seems to be not needed.
890
891=== poll, select ===
892
893TBD.  Seems to be not needed.
894
895=== RMAN(9) (Resource management) ===
896
897http://www.freebsd.org/cgi/man.cgi?query=rman
898
899TBD.  Seems to be not needed.
900
901=== DEVCLASS(9), DEVICE(9), DRIVER(9), MAKE_DEV(9) (Device management) ===
902
903http://www.freebsd.org/cgi/man.cgi?query=devclass
904
905http://www.freebsd.org/cgi/man.cgi?query=device
906
907http://www.freebsd.org/cgi/man.cgi?query=driver
908
909http://www.freebsd.org/cgi/man.cgi?query=make_dev
910
911Use FreeBSD implementation as far as possible.  FreeBSD has a nice API for
912dynamic device handling.  It may be interesting for RTEMS to use this API
913internally in the future.
914
915=== BUS_SPACE(9), BUS_DMA(9) (Bus and DMA access) ===
916
917http://www.freebsd.org/cgi/man.cgi?query=bus_space
918
919http://www.freebsd.org/cgi/man.cgi?query=bus_dma
920
921Likely BSP dependent.  A default implementation for memory mapped linear access
922is easy to provide.  The current heap implementation supports all properties
923demanded by bus_dma (including the boundary constraint).
924
925== RTEMS Replacements by File Description ==
926
927Note:  Files with a status of USB are used by the USB test and have at least
928been partially tested.  If they contain both USB and Nic, then they are used
929by both and MAY contain methods that have not been tested yet.  Files that
930are only used by the Nic test are the most suspect.
931
932----
933rtems-libbsd File:      rtems-bsd-assert.c
934FreeBSD File:           rtems-bsd-config.h redefines BSD_ASSERT.
935Description:            This file contains the support method rtems_bsd_assert_func().
936Status:                 USB, Nic
937
938rtems-libbsd File:      rtems-bsd-autoconf.c
939FreeBSD File:           FreeBSD has BSP specific autoconf.c
940Description:            This file contains configuration methods that are used to setup the system.
941Status:                 USB
942
943rtems-libbsd File:      rtems-bsd-bus-dma.c
944FreeBSD File:           FreeBSD has BSP specific busdma_machdep.c
945Description:
946Status:                 USB, Nic
947
948rtems-libbsd File:      rtems-bsd-bus-dma-mbuf.c
949FreeBSD File:           FreeBSD has BSP specific busdma_machdep.c
950Description:
951Status:                 Nic
952
953rtems-libbsd File:      rtems-bsd-callout.c
954FreeBSD File:           kern/kern_timeout.c
955Description:
956Status:                 USB, Nic
957
958rtems-libbsd File:      rtems-bsd-cam.c
959FreeBSD File:           cam/cam_sim.c
960Description:
961Status:                 USB
962
963rtems-libbsd File:      rtems-bsd-condvar.c
964FreeBSD File:           kern/kern_condvar.c
965Description:
966Status:                 USB
967
968rtems-libbsd File:      rtems-bsd-copyinout.c
969FreeBSD File:           bsp specific copyinout.c )
970Description:            Note: The FreeBSD file is split with some methods being in rtems-bsd-support
971Status:                 Nic
972
973rtems-libbsd File:      rtems-bsd-delay.c
974FreeBSD File:           bsp specific file with multiple names
975Description:
976Status:                 USB, Nic
977
978rtems-libbsd File:      rtems-bsd-descrip.c
979FreeBSD File:           kern/kern_descrip.c
980Description:
981Status:                 Nic
982
983rtems-libbsd File:      rtems-bsd-generic.c
984FreeBSD File:           kern/sys_generic.c
985Description:
986Status:                 Nic
987
988rtems-libbsd File:      rtems-bsd-init.c
989FreeBSD File:           N/A
990Description:
991Status:                 USB, Nic
992
993rtems-libbsd File:      rtems-bsd-init-with-irq.c
994FreeBSD File:           N/A
995Description:
996Status:                 USB, Nic
997
998rtems-libbsd File:      rtems-bsd-jail.c
999FreeBSD File:           kern/kern_jail.c
1000Description:
1001Status:                 USB, Nic
1002
1003rtems-libbsd File:      rtems-bsd-lock.c
1004FreeBSD File:           kern/subr_lock.c
1005Description:
1006Status:                 USB, Nic
1007
1008rtems-libbsd File:      rtems-bsd-log.c
1009FreeBSD File:           kern/subr_prf.c
1010Description:
1011Status:                 Nic
1012
1013rtems-libbsd File:      rtems-bsd-malloc.c
1014FreeBSD File:           kern/kern_malloc.c
1015Description:
1016Status:                 USB, Nic
1017
1018rtems-libbsd File:      rtems-bsd-mutex.c
1019FreeBSD File:           kern/kern_mutex.c
1020Description:
1021Status:                 USB, Nic
1022
1023rtems-libbsd File:      rtems-bsd-newproc.c
1024FreeBSD File:           N/A
1025Description:
1026Status:                 Nic
1027
1028rtems-libbsd File:      rtems-bsd-nexus.c
1029FreeBSD File:           bsp specific nexus.c
1030Description:
1031Status:                 USB
1032
1033rtems-libbsd File:      rtems-bsd-panic.c
1034FreeBSD File:           boot/common/panic.c
1035Description:
1036Status:                 USB, Nic
1037
1038rtems-libbsd File:      rtems-bsd-rwlock.c
1039FreeBSD File:           kern_rwlock.c
1040Description:
1041Status:                 USB, Nic
1042
1043rtems-libbsd File:      rtems-bsd-shell.c
1044FreeBSD File:           N/A
1045Description:
1046Status:                 USB
1047
1048rtems-libbsd File:      rtems-bsd-signal.c
1049FreeBSD File:           kern/kern_sig.c
1050Description:
1051Status:                 Nic
1052
1053rtems-libbsd File:      rtems-bsd-smp.c
1054FreeBSD File:           N/A
1055Description:
1056Status:                 Nic
1057
1058rtems-libbsd File:      rtems-bsd-support.c
1059FreeBSD File:           bsp specific copyinout.c
1060Description:            Note: the FreeBSD file is split with some methods being in rtems-bsd-copyinout.
1061Status:                 USB, Nic
1062
1063rtems-libbsd File:      rtems-bsd-sx.c
1064FreeBSD File:           kern/kern_sx.c
1065Description:            Status: USB, Nic
1066
1067rtems-libbsd File:      rtems-bsd-synch.c
1068FreeBSD File:           kern/kern_synch.c
1069Description:
1070Status:                 USB, Nic
1071
1072rtems-libbsd File:      rtems-bsd-syscalls.c
1073FreeBSD File:           User API for kern/uipc_syscalls.c
1074Description:
1075Status:                 Nic
1076
1077rtems-libbsd File:      rtems-bsd-sysctlbyname.c
1078FreeBSD File:           User API for sysctlbyname(3)
1079Description:
1080Status:
1081
1082rtems-libbsd File:      rtems-bsd-sysctl.c
1083FreeBSD File:           User API for sysctl(8)
1084Description:
1085Status:
1086
1087rtems-libbsd File:      rtems-bsd-sysctlnametomib.c
1088FreeBSD File:           User API for sysctlnametomib
1089Description:
1090Status:
1091
1092rtems-libbsd File:      rtems-bsd-taskqueue.c
1093FreeBSD File:           kern/subr_taskqueue.c
1094Description:
1095Status:                 Nic
1096
1097rtems-libbsd File:      rtems-bsd-thread.c
1098FreeBSD File:           kern/kern_kthread.c
1099Description:
1100Status:                 USB, Nic
1101
1102rtems-libbsd File:      rtems-bsd-timeout.c
1103FreeBSD File:           kern/kern_timeout.c
1104Description:
1105Status:                 Nic
1106
1107rtems-libbsd File:      rtems-bsd-timesupport.c
1108FreeBSD File:           kern/kern_clock.c
1109Description:
1110Status:                 Nic
1111
1112rtems-libbsd File:      rtems-bsd-vm_glue.c
1113FreeBSD File:           vm/vm_glue.c
1114Description:
1115Status:                 USB, Nic
1116----
1117
1118== Notes by File ==
1119
1120altq_subr.c - Arbitrary choices were made in this file that RTEMS would
1121not support tsc frequency change.  Additionally, the clock frequency
1122for machclk_freq is always measured for RTEMS.
1123
1124conf.h - In order to add make_dev and destroy_dev, variables in the cdev
1125structure that were not being used were conditionally compiled out. The
1126capability of supporting children did not appear to be needed and was
1127not implemented in the rtems version of these routines.
1128
1129== NICs Status ==
1130
1131----
1132Driver                  Symbol                          Status
1133======                  ======                          ======
1134RealTek                 _bsd_re_pcimodule_sys_init      Links
1135EtherExpress            _bsd_fxp_pcimodule_sys_init     Links
1136DEC tulip               _bsd_dc_pcimodule_sys_init      Links
1137Broadcom BCM57xxx       _bsd_bce_pcimodule_sys_init     Links
1138Broadcom BCM4401        _bsd_bfe_pcimodule_sys_init     Links
1139Broadcom BCM570x        _bsd_bge_pcimodule_sys_init     Needs Symbols (A)
1140E1000 IGB               _bsd_igb_pcimodule_sys_init     Links
1141E1000 EM                _bsd_em_pcimodule_sys_init      Links
1142Cadence                 ?                               Links, works.
1143----
1144
1145To add a NIC edit rtemsbsd/include/bsp/nexus-devices.h and add the driver
1146reference to the architecture and/or BSP. For example to add the RealTek driver
1147add:
1148
1149SYSINIT_DRIVER_REFERENCE(re, pci);
1150
1151and to add the MII PHY driver add:
1152
1153SYSINIT_DRIVER_REFERENCE(rge, miibus);
1154
1155The PC BSP has these entries.
1156
1157Symbols (A)
1158         pci_get_vpd_ident
1159
1160=== Cadence ===
1161
1162The cadence driver works on the Xilinx Zynq platform. The hardware checksum
1163support works on real hardware but does not seem to be supported on qemu
1164therefore the default state is to disable TXCSUM and RXCSUM and this can be
1165enabled from the shell with:
1166
1167  # ifconfig cgem0 rxcsum txcsum
1168
1169or with an ioctl call to the network interface driver with SIOCSIFCAP and the
1170mask IFCAP_TXCSUM and IFCAP_RXCSUM set.
1171
1172== Problems to report to FreeBSD ==
1173
1174The MMAP_NOT_AVAILABLE define is inverted on its usage.  When it is
1175defined the mmap method is called. Additionally, it is not used
1176thoroughly. It is not used in the unmap portion of the source.
1177The file rec_open.c uses the define MMAP_NOT_AVAILABLE to wrap
1178the call to mmap and file rec_close.c uses the munmap method.
Note: See TracBrowser for help on using the repository browser.