[e3e88e1] | 1 | Guidelines for Developing and Contributing Code |
---|
| 2 | =============================================== |
---|
| 3 | |
---|
| 4 | Introduction |
---|
| 5 | ------------ |
---|
| 6 | |
---|
| 7 | This guide aims to help developing and contributing code to the libbsd. One |
---|
| 8 | goal of the libbsd is to stay in synchronization with FreeBSD. This is only |
---|
| 9 | feasible if certain rules are in place. Otherwise, managing more than a |
---|
| 10 | thousand imported source files will become too labour intensive eventually. |
---|
| 11 | |
---|
| 12 | What is in the Git Repository |
---|
| 13 | ----------------------------- |
---|
| 14 | |
---|
| 15 | The libbsd a self-contained kit with FreeBSD and RTEMS components pre-merged. |
---|
| 16 | The Waf wscript in libbsd is automatically generated. |
---|
| 17 | |
---|
| 18 | Any changes to source in the `freebsd` directories will need to be merged |
---|
| 19 | upstream into our master FreeBSD checkout, the `freebsd-org` submodule. |
---|
| 20 | |
---|
| 21 | The repository contains two FreeBSD source trees. In the `freebsd` directory |
---|
| 22 | are the so called *managed* FreeBSD sources used to build the BSD library. The |
---|
| 23 | FreeBSD source in `freebsd-org` is the *master* version. The |
---|
| 24 | `freebsd-to-rtems.py` script is used to transfer files between the two trees. |
---|
| 25 | In general terms, if you have modified managed FreeBSD sources, you will need |
---|
| 26 | to run the script in *revert* or *reverse* mode using the `-R` switch. This |
---|
| 27 | will copy the source back to your local copy of the master FreeBSD source so |
---|
| 28 | you can run `git diff` against the upstream FreeBSD source. If you want to |
---|
| 29 | transfer source files from the master FreeBSD source to the manged FreeBSD |
---|
| 30 | sources, then you must run the script in *forward* mode (the default). |
---|
| 31 | |
---|
| 32 | Organization |
---|
| 33 | ------------ |
---|
| 34 | |
---|
| 35 | The top level directory contains a few directories and files. The following |
---|
| 36 | are important to understand |
---|
| 37 | |
---|
| 38 | * `freebsd-to-rtems.py` - script to convert to and free FreeBSD and RTEMS trees, |
---|
| 39 | * `create-kernel-namespace.sh` - script to create the kernel namespace header <machine/rtems-bsd-kernel-namespace.h, |
---|
| 40 | * `wscript` - automatically generated, |
---|
| 41 | * `freebsd/` - from FreeBSD by script, |
---|
| 42 | * `rtemsbsd/` - RTEMS specific implementations of FreeBSD kernel support routines, |
---|
| 43 | * `testsuite/` - RTEMS specific tests, and |
---|
| 44 | * `libbsd.txt` - documentation in Asciidoc. |
---|
| 45 | |
---|
| 46 | Moving Code Between Managed and Master FreeBSD Source |
---|
| 47 | ----------------------------------------------------- |
---|
| 48 | |
---|
| 49 | The script `freebsd-to-rtems.py` is used to copy code from FreeBSD to the |
---|
| 50 | rtems-libbsd tree and to reverse this process. This script attempts to |
---|
| 51 | automate this process as much as possible and performs some transformations |
---|
| 52 | on the FreeBSD code. Its command line arguments are shown below: |
---|
| 53 | |
---|
| 54 | ``` |
---|
| 55 | freebsd-to-rtems.py [args] |
---|
| 56 | -?|-h|--help print this and exit |
---|
| 57 | -d|--dry-run run program but no modifications |
---|
| 58 | -D|--diff provide diff of files between trees |
---|
| 59 | -e|--early-exit evaluate arguments, print results, and exit |
---|
| 60 | -m|--makefile Warning: depreciated and will be removed |
---|
| 61 | -b|--buildscripts just generate the build scripts |
---|
| 62 | -S|--stats Print a statistics report |
---|
| 63 | -R|--reverse default FreeBSD -> RTEMS, reverse that |
---|
| 64 | -r|--rtems RTEMS Libbsd directory (default: '.') |
---|
| 65 | -f|--freebsd FreeBSD SVN directory (default: 'freebsd-org') |
---|
| 66 | -v|--verbose enable verbose output mode |
---|
| 67 | ``` |
---|
| 68 | |
---|
| 69 | In its default mode of operation, freebsd-to-rtems.py is used to copy code |
---|
| 70 | from FreeBSD to the rtems-libbsd tree and perform transformations. In forward |
---|
| 71 | mode, the script may be requested to just generate the Waf script. |
---|
| 72 | |
---|
| 73 | In *reverse mode*, this script undoes those transformations and copies |
---|
| 74 | the source code back to the *master* FreeBSD tree. This allows us to do |
---|
| 75 | 'git diff', evaluate changes made by the RTEMS Project, and report changes |
---|
| 76 | back to FreeBSD upstream. |
---|
| 77 | |
---|
| 78 | In either mode, the script may be asked to perform a dry-run or be verbose. |
---|
| 79 | Also, in either mode, the script is also smart enough to avoid copying over |
---|
| 80 | files which have not changed. This means that the timestamps of files are |
---|
| 81 | not changed unless the contents change. The script will also report the |
---|
| 82 | number of files which changed. In verbose mode, the script will print |
---|
| 83 | the name of the files which are changed. |
---|
| 84 | |
---|
| 85 | To add or update files in the RTEMS FreeBSD tree first run the *reverse mode* |
---|
| 86 | and move the current set of patches FreeBSD. The script may warn you if a file |
---|
| 87 | is not present at the destination for the direction. This can happen as files |
---|
| 88 | not avaliable at the FreeBSD snapshot point have been specially added to the |
---|
| 89 | RTEMS FreeBSD tree. Warnings can also appear if you have changed the list of |
---|
| 90 | files in libbsd.py. The reverse mode will result in the FreeBSD having |
---|
| 91 | uncommitted changes. You can ignore these. Once the reverse process has |
---|
| 92 | finished edit libbsd.py and add any new files then run the forwad mode to bring |
---|
| 93 | those files into the RTEMS FreeBSD tree. |
---|
| 94 | |
---|
| 95 | The following is an example forward run with no changes. |
---|
| 96 | |
---|
| 97 | ``` |
---|
| 98 | $ ./freebsd-to-rtems.py -v |
---|
| 99 | Verbose: yes (1) |
---|
| 100 | Dry Run: no |
---|
| 101 | Diff Mode Enabled: no |
---|
| 102 | Only Generate Build Scripts: no |
---|
| 103 | RTEMS Libbsd Directory: . |
---|
| 104 | FreeBSD SVN Directory: freebsd-org |
---|
| 105 | Direction: forward |
---|
| 106 | Forward from FreeBSD GIT into . |
---|
| 107 | 0 file(s) were changed: |
---|
| 108 | ``` |
---|
| 109 | |
---|
| 110 | The script may also be used to generate a diff in either forward or reverse |
---|
| 111 | direction. |
---|
| 112 | |
---|
| 113 | You can add more than one verbose option (-v) to the command line and get more |
---|
| 114 | detail and debug level information from the command. |
---|
| 115 | |
---|
| 116 | FreeBSD Version of Imported Files and Directories |
---|
| 117 | ------------------------------------------------- |
---|
| 118 | |
---|
| 119 | * *, trunk, 2017-04-04, 642b174daddbd0efd9bb5f242c43f4ab4db6869f. |
---|
| 120 | |
---|
| 121 | How to import code from FreeBSD |
---|
| 122 | ------------------------------- |
---|
| 123 | |
---|
| 124 | * In case you import files from a special FreeBSD version, then update the list above. |
---|
| 125 | * Run `git status` and make sure your working directory is clean. |
---|
| 126 | * Run `./freebsd-to-rtems.py -R` |
---|
| 127 | * Run `./freebsd-to-rtems.py` |
---|
| 128 | * Run `git status` and make sure your working directory is clean. If you see modified files, then the `freebsd-to-rtems.py` script needs to be fixed first. |
---|
| 129 | * Add the files to import to `libbsd.py`. |
---|
| 130 | * Run `./freebsd-to-rtems.py` |
---|
| 131 | * Immediately check in the imported files without the changes to `libbsd_waf.py`. Do not touch the imported files yourself at this point. |
---|
| 132 | * Port the imported files to RTEMS. See 'Rules for Modifying FreeBSD Source'. |
---|
| 133 | * Add a test to the testsuite if possible. |
---|
| 134 | * Run `./create-kernel-namespace.sh` if you imported kernel space headers. Add only your new defines via `git add -p rtemsbsd/include/machine/rtems-bsd-kernel-namespace.h`. |
---|
| 135 | * Create one commit from this. |
---|
| 136 | |
---|
| 137 | The -S or --stats option generates reports the changes we have made to |
---|
| 138 | FreeBSD. If the code has been reserved into the original FreeBSD tree it will |
---|
| 139 | show nothing has changed. To see what we have change: |
---|
| 140 | |
---|
| 141 | ``` |
---|
| 142 | $ cd freebsd-org |
---|
| 143 | $ git checkout -- . |
---|
| 144 | $ cd .. |
---|
| 145 | $ ./freebsd-to-rtems.py -R -S -d |
---|
| 146 | ``` |
---|
| 147 | |
---|
| 148 | The report lists the files change based on the opacity level. The opacity is a |
---|
| 149 | measure on how much of a file differs from the original FreeBSD source. The |
---|
| 150 | lower the value the more transparent the source file it. |
---|
| 151 | |
---|
| 152 | Porting of User-Space Utilities |
---|
| 153 | ------------------------------ |
---|
| 154 | |
---|
| 155 | The theory behind the described method is to put all BSS and initialized data |
---|
| 156 | objects into a named section. This section then will be saved before the code is |
---|
| 157 | executed and restored after it has finished. This method limits to a single |
---|
| 158 | threaded execution of the application but minimizes the necessary changes to the |
---|
| 159 | original FreeBSD code. |
---|
| 160 | |
---|
| 161 | * Import and commit the unchanged source files like described above. |
---|
[cd7d52d] | 162 | * Add the files to the [libbsd.py](libbsd.py) and build them. |
---|
[e3e88e1] | 163 | * Check the sources for everything that can be made const. This type of patches |
---|
| 164 | should go back to the upstream FreeBSD sources. |
---|
| 165 | * Move static variables out of functions if necessary (search for |
---|
[cd7d52d] | 166 | "\tstatic"). These patches most likely will not be accepted into FreeBSD. |
---|
[e3e88e1] | 167 | * Add a rtems_bsd_command_PROGNAME() wrapper function to the source file |
---|
| 168 | containing the main function (e.g. PROGNAME = pfctl). For an example look at |
---|
[cd7d52d] | 169 | `rtems_bsd_command_pfctl()` in [pfctl.c](freebsd/sbin/pfctl/pfctl.c). |
---|
[e3e88e1] | 170 | * You probably have to use getopt_r() instead of getopt(). Have a look at |
---|
[cd7d52d] | 171 | [pfctl.c](freebsd/sbin/pfctl/pfctl.c). |
---|
[e3e88e1] | 172 | * Build the libbsd without optimization. |
---|
| 173 | * Use the `userspace-header-gen.py` to generate some necessary header |
---|
| 174 | files. It will generate one `rtems-bsd-PROGNAME-MODULE-data.h` per object file, one |
---|
| 175 | `rtems-bsd-PROGNAME-namespace.h` and one `rtems-bsd-PROGNAME-data.h`. To call |
---|
| 176 | the script, you have to compile the objects and afterwards run the helper |
---|
| 177 | script with a call similar to this one: |
---|
| 178 | `python ./userspace-header-gen.py build/arm-rtems4.12-xilinx_zynq_a9_qemu/freebsd/sbin/pfctl/*.o -p pfctl` |
---|
| 179 | Replace the name (given via -p option) by the name of the userspace tool. It |
---|
| 180 | has to match the name that is used in the RTEMS linker set further below. |
---|
| 181 | * If you regenerated files that have already been generated, you may have to |
---|
| 182 | remove RTEMS-specific names from the namespace. The defaults (linker set names |
---|
[cd7d52d] | 183 | and rtems_bsd_program_.*) should already be filtered. |
---|
[e3e88e1] | 184 | * Put the generated header files into the same folder like the source files. |
---|
[cd7d52d] | 185 | * At the top of each source file place the following right after the user-space header: |
---|
| 186 | ```c |
---|
| 187 | #ifdef __rtems__ |
---|
| 188 | #include <machine/rtems-bsd-program.h> |
---|
| 189 | #include "rtems-bsd-PROGNAME-namespace.h" |
---|
| 190 | #endif /* __rtems__ */ |
---|
| 191 | ``` |
---|
| 192 | The following command may be useful: |
---|
| 193 | ``` |
---|
| 194 | sed -i 's%#include <machine/rtems-bsd-user-space.h>%#include <machine/rtems-bsd-user-space.h>\n#ifdef __rtems__\n#include <machine/rtems-bsd-program.h>\n#include "rtems-bsd-PROGNAME-namespace.h"\n#endif /* __rtems__ */%' *.c |
---|
| 195 | ``` |
---|
| 196 | * At the bottom of each source file place the follwing: |
---|
| 197 | ```c |
---|
| 198 | #ifdef __rtems__ |
---|
| 199 | #include "rtems-bsd-PROGNAME-FILE-data.h" |
---|
| 200 | #endif /* __rtems__ */ |
---|
| 201 | ``` |
---|
| 202 | The following command may be useful: |
---|
| 203 | ``` |
---|
| 204 | for i in *.c ; do n=$(basename $i .c) ; echo -e "#ifdef __rtems__\n#include \"rtems-bsd-PROGNAME-$n-data.h\"\n#endif /* __rtems__ */" >> $i ; done |
---|
| 205 | ``` |
---|
[e3e88e1] | 206 | * Create one compilable commit. |
---|
| 207 | |
---|
| 208 | Rules for Modifying FreeBSD Source |
---|
| 209 | ---------------------------------- |
---|
| 210 | |
---|
| 211 | Changes in FreeBSD files must be done using `__rtems__` C pre-processor guards. |
---|
| 212 | This makes synchronization with the FreeBSD upstream easier and is very |
---|
| 213 | important. Patches which do not follow these rules will be rejected. Only add |
---|
| 214 | lines. If your patch contains lines starting with a `-`, then this is wrong. |
---|
| 215 | Subtract code by added `#ifndef __rtems__`. For example: |
---|
| 216 | |
---|
| 217 | ```c |
---|
| 218 | /* Global variables for the kernel. */ |
---|
| 219 | |
---|
| 220 | #ifndef __rtems__ |
---|
| 221 | /* 1.1 */ |
---|
| 222 | extern char kernelname[MAXPATHLEN]; |
---|
| 223 | #endif /* __rtems__ */ |
---|
| 224 | |
---|
| 225 | extern int tick; /* usec per tick (1000000 / hz) */ |
---|
| 226 | ``` |
---|
| 227 | |
---|
| 228 | ```c |
---|
| 229 | #if defined(_KERNEL) || defined(_WANT_FILE) |
---|
| 230 | #ifdef __rtems__ |
---|
| 231 | #include <rtems/libio_.h> |
---|
| 232 | #include <sys/fcntl.h> |
---|
| 233 | #endif /* __rtems__ */ |
---|
| 234 | /* |
---|
| 235 | * Kernel descriptor table. |
---|
| 236 | * One entry for each open kernel vnode and socket. |
---|
| 237 | * |
---|
| 238 | * Below is the list of locks that protects members in struct file. |
---|
| 239 | * |
---|
| 240 | * (f) protected with mtx_lock(mtx_pool_find(fp)) |
---|
| 241 | * (d) cdevpriv_mtx |
---|
| 242 | * none not locked |
---|
| 243 | */ |
---|
| 244 | ``` |
---|
| 245 | |
---|
| 246 | ```c |
---|
| 247 | extern int profprocs; /* number of process's profiling */ |
---|
| 248 | #ifndef __rtems__ |
---|
| 249 | extern volatile int ticks; |
---|
| 250 | #else /* __rtems__ */ |
---|
| 251 | #include <rtems/score/watchdogimpl.h> |
---|
| 252 | #define ticks _Watchdog_Ticks_since_boot |
---|
| 253 | #endif /* __rtems__ */ |
---|
| 254 | |
---|
| 255 | #endif /* _KERNEL */ |
---|
| 256 | ``` |
---|
| 257 | |
---|
| 258 | Add nothing (even blank lines) before or after the `__rtems__` guards. Always |
---|
| 259 | include a `__rtems__` in the guards to make searches easy, so use |
---|
| 260 | |
---|
| 261 | * `#ifndef __rtems__`, |
---|
| 262 | * `#ifdef __rtems__`, |
---|
| 263 | * `#else /* __rtems__ */`, and |
---|
| 264 | * `#endif /* __rtems__ */`. |
---|
| 265 | |
---|
| 266 | The guards must start at the begin of the line. Examples for wrong guards: |
---|
| 267 | |
---|
| 268 | ```c |
---|
| 269 | static void |
---|
| 270 | guards_must_start_at_the_begin_of_the_line(int j) |
---|
| 271 | { |
---|
| 272 | |
---|
| 273 | /* WRONG */ |
---|
| 274 | #ifdef __rtems__ |
---|
| 275 | return (j + 1); |
---|
| 276 | #else /* __rtems__ */ |
---|
| 277 | return (j + 2); |
---|
| 278 | #endif /* __rtems__ */ |
---|
| 279 | } |
---|
| 280 | |
---|
| 281 | static void |
---|
| 282 | missing_rtems_comments_in_the_guards(int j) |
---|
| 283 | { |
---|
| 284 | |
---|
| 285 | #ifdef __rtems__ |
---|
| 286 | return (j + 3); |
---|
| 287 | /* WRONG */ |
---|
| 288 | #else |
---|
| 289 | return (j + 4); |
---|
| 290 | #endif |
---|
| 291 | } |
---|
| 292 | ``` |
---|
| 293 | |
---|
| 294 | The FreeBSD build and configuration system uses option header files, e.g. |
---|
| 295 | `#include "opt_xyz.h"` in an unmodified FreeBSD file. This include is |
---|
| 296 | transformed by the import script into `#include <rtems/bsd/local/opt_xyz.h>`. Do |
---|
| 297 | not disable option header includes via guards. Instead, add an empty option |
---|
| 298 | header, e.g. `touch rtemsbsd/include/rtems/bsd/local/opt_xyz.h`. |
---|
| 299 | ```c |
---|
| 300 | /* WRONG */ |
---|
| 301 | #ifndef __rtems__ |
---|
| 302 | #include <rtems/bsd/local/opt_xyz.h> |
---|
| 303 | #endif /* __rtems__ */ |
---|
| 304 | ``` |
---|
| 305 | |
---|
| 306 | In general, provide empty header files and do not guard includes. |
---|
| 307 | |
---|
| 308 | For new code use |
---|
| 309 | [STYLE(9)](http://www.freebsd.org/cgi/man.cgi?query=style&apropos=0&sektion=9). |
---|
| 310 | |
---|
| 311 | Do not format original FreeBSD code. Do not perform white space changes even |
---|
| 312 | if you get git commit warnings. |
---|
| 313 | |
---|
| 314 | Automatically Generated FreeBSD Files |
---|
| 315 | ------------------------------------- |
---|
| 316 | |
---|
| 317 | Some source and header files are automatically generated during the FreeBSD |
---|
| 318 | build process. The `Makefile.todo` file performs this manually. The should be |
---|
| 319 | included in `freebsd-to-rtems.py` script some time in the future. For details, |
---|
| 320 | see also |
---|
| 321 | [KOBJ(9)](http://www.freebsd.org/cgi/man.cgi?query=kobj&sektion=9&apropos=0). |
---|