New build system

[Sebastian Huber]

Replace the existing Autoconf and Automake based build system with something new. Goals:

• Easier configuration of BSPs (BSP options)

• Easier to maintain (nobody understands the details of the existing build system)

• Faster builds (especially on Windows)

• Reusable build specifications (e.g. generate documentation for BSP options for the user manual)

• Validation of built artefacts (e.g. ensure that the objects are built as specified using the DWARF debug information)

• Support building of BSPs external to the project

• Customization of the build (e.g. build only a subset of the RTEMS functions)

• Support alternative compilers such as clang instead of GCC

Proposed solution:

• Use waf for the build system

• Specify the build through build specification items maintained by Doorstop

[Sebastian Huber <sebastian.huber@…>]

In 9f3c558/rtems:

tests: Remove superfluous SMPTESTS define

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f32e119/rtems:

psxtests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 150fbaf9/rtems:

psxtmtests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In a467f59/rtems:

sptests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 51765553/rtems:

sptests: Remove obsolete semaphore consume driver

This driver is no longer used by a test program.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 5ba4f383/rtems:

libtests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In c89a5dc8/rtems:

samples: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 34cbac1/rtems:

tmtests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In eae263c/rtems:

mptests/mp14: Make MAX_LONG_TEST_DURATION constant

If there is a real need, it can be made a configuration option again.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 786ae3f/rtems:

ada/samples: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 0b520cb/rtems:

mptests: Avoid build system defined defines

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In b88c7583/rtems:

fstests: Use tmacros.h instead of pmacros.h

This avoids an extra include path.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 37d509b/rtems:

libtests: Remove superfluous include path

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 6da85f49/rtems:

block08: Use local include

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 1e343825/rtems:

tests: Simplify fatal error test support

Move system.h to shared init.c.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 1a48cbf/rtems:

sptests: Avoid include path magic

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f515948/rtems:

ada/mptests: Make them compile clean

Fix all warnings.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 984d7443/rtems:

bsp/gumstix: Remove ON_SKYEYE Automake conditional

It is fine to build the drivers always.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In bc28b65/rtems:

bsp/beagle: Rename linker command file

This BSP family uses only one linker command file. Use the standard
name.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 41ab50e4/rtems:

bsp/t32mppc: Rename linker command file

This BSP family uses only one linker command file. Use the standard
name.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 9c601b00/rtems:

bsp/i386: Remove unused BSP_HAS_SMP

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In b882b07/rtems:

Remove BSP_SMALL_MEMORY BSP option

Use the test state configuration instead.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 75d9e79f/rtems:

bsp/lpc32xx: Remove unused TESTS_USE_PRINTK

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In af321aa/rtems:

bsp/altcycv_devkit: Rename linker command file

This BSP family uses only one linker command file. Use the standard
name.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In a52d6174/rtems:

bsp/imx7: Rename linker command file

This BSP family uses only one linker command file. Use the standard
name.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 38207a31/rtems:

Move feature macro before "config.h" include

This allows to use header includes in "config.h" to reduce the build
configuration checks.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f377998/rtems:

imfs: Add IMFS_make_linfile()

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 3e9f5a55/rtems:

bsp/gen83xx: Remove obsolete linker command file

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 05a32e15/rtems:

bsp/mpc55xxevb: Remove obsolete linker cmd file

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In a0304e8d/rtems:

libtest: Output basename of source files

Output only the basename of source files to be independent of the build
system source paths.

In the future it may be better to use the GCC -fmacro-prefix-map option.
This option is available in GCC 8 and later. It is not yet available in
clang.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 273e8b7/rtems:

libtests/tar0[12]:: Use static archive content

This simplifies the build process. Do not generate the archive content
through the build system. Let the version control system deal with
symbolic links.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 8f021bd/rtems:

libtests/dl*: Do not generate files via "echo"

Add the static files to the repository. This simplifies the build.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 5c2e7104/rtems:

libtests: Use '-' for TAR file names

Use uniform pattern for all TAR file names. Use the dl* tests as a
template.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 4a05652/rtems:

libtests/dl*: Rename source files

Rename source files to use a %.c -> %.o and %.cc -> %.o pattern. Use
*.cc for C++ source files instead of *.cpp to be in line with other C++
source files.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 1fca1665/rtems:

testsuites/ada: Introduce init.c files

Provide an init.c for each Ada test instead of the #include <config.h>
magic which works only due to hand crafted include paths.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In ae716da/rtems:

ada/sp19: Add m4 generated sptest.adb

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 8f6b7abd/rtems:

Move feature macro before "config.h" include

This allows to use header includes in "config.h" to reduce the build
configuration checks.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In a6879a4/rtems:

testsuites: Remove rtems_test_pause*()

The rtems_test_pause() and rtems_test_pause_and_screen_number() macros
had different implementations depending on the RTEMS_TEST_NO_PAUSE
define. This define was defined to 1 by default. The user was able to
change this via the undocumented --disable-test-no-pause configure
command line option.

Pausing tests and waiting for user input contradicts the goal of having
automated test runs. Remove this feature.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 1380c822/rtems:

bsp/xen: Use BSP options for all linkcmds vars

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f30dd1f/rtems:

mptests: Fix configuration

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 449fc74/rtems-docs:

eng: Add BSP build system chapter

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 4ea43e3/rtems-docs:

user: Document BSP new build system

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f3f0370f/rtems:

build: Alternative build system based on waf

Update #3818.

[kgardas]

with new build system smp enabled build of pc686 fails. Linkage of samples fails with

/export/home/karel/sfw/rtems/6-tools-core2/lib/gcc/i386-rtems6/10.2.1/../../../../i386-rtems6/bin/ld: ./librtemsbsp.a(smp-imps.c.10.o): in function `boot_cpu':
/export/home/karel/git/rtems/rtems-head/build/i386/pc686/../../../bsps/i386/pc386/start/smp-imps.c:274: undefined reference to `_binary_appstart_bin_start'
/export/home/karel/sfw/rtems/6-tools-core2/lib/gcc/i386-rtems6/10.2.1/../../../../i386-rtems6/bin/ld: /export/home/karel/git/rtems/rtems-head/build/i386/pc686/../../../bsps/i386/pc386/start/smp-imps.c:274: undefined reference to `_binary_appstart_bin_size'
collect2: error: ld returned 1 exit status

that's due to fact this part of Makefile based build was not translated to the new build system:

appcpustart.$(OBJEXT): ../../../../../../bsps/i386/pc386/start/startAP.S
        $(CPPASCOMPILE) $(AM_CPPFLAGS) -o $@ -c $<

appstart.$(OBJEXT): appcpustart.$(OBJEXT)
        $(LD) -N  \
            -Ttext 0x70000 -e app_processor_start -nostdlib \
            -o appstart_tmp.exe $<
        $(OBJCOPY) -O binary appstart_tmp.exe appstart.bin
        $(OBJCOPY) -I binary -O elf32-i386 -B i386 appstart.bin $@

besides this there is also error in files (getcpuid.c seems to be gone, bspsmp.c needs to be there:

diff --git a/spec/build/bsps/i386/pc386/objsmp.yml b/spec/build/bsps/i386/pc386/objsmp.yml
index af882cb425..9e83bd7a01 100644
--- a/spec/build/bsps/i386/pc386/objsmp.yml
+++ b/spec/build/bsps/i386/pc386/objsmp.yml
@@ -11,6 +11,7 @@ includes: []
 install: []
 links: []
 source:
-- bsps/i386/pc386/start/getcpuid.c
+#- bsps/i386/pc386/start/getcpuid.c
+- bsps/i386/pc386/start/bspsmp.c
 - bsps/i386/pc386/start/smp-imps.c
 type: build

unfortunately so far I struggle with translating this to waf, probably should look for some waf doc or so. E.g. linking steps of appcpustart.o into appstart.o is what makes me trouble. Also I'm not able to find out how rtemsbsp lib is linked and its dependency specified...

[Sebastian Huber <sebastian.huber@…>]

In 127c3ea5/rtems:

build: Fix i386/pc386 with SMP enabled

Update #3818.

[Sebastian Huber]

Replying to kgardas:

unfortunately so far I struggle with translating this to waf, probably should look for some waf doc or so. E.g. linking steps of appcpustart.o into appstart.o is what makes me trouble. Also I'm not able to find out how rtemsbsp lib is linked and its dependency specified...

Thanks testing the new build system and for the bug report. I checked in a fix, could you please test if this fixes the issues on this BSP.

Working with waf is also not easy for me. Fortunately, we don't have that many areas in the build which need a special treatment like this. It was the first spot which need to include a specially build object file in a library.

[kgardas]

Replying to Sebastian Huber:

Replying to kgardas:

unfortunately so far I struggle with translating this to waf, probably should look for some waf doc or so. E.g. linking steps of appcpustart.o into appstart.o is what makes me trouble. Also I'm not able to find out how rtemsbsp lib is linked and its dependency specified...

Thanks testing the new build system and for the bug report. I checked in a fix, could you please test if this fixes the issues on this BSP.

Working with waf is also not easy for me. Fortunately, we don't have that many areas in the build which need a special treatment like this. It was the first spot which needed to include a specially build object file in a library.

Build builds both no-SMP and SMP BSPs fine, but neither of them runs on neither qemu nor on real hardware. If you find helpful, then qemu connected gdb shows those 3 traces over few seconds of BSP run:

Program received signal SIGINT, Interrupt.
0x00008d54 in _RBTree_Insert_inline (less=<optimized out>, key=<synthetic pointer>, the_node=0x5ccc3d04, the_rbtree=0x736f4d74) at ../../../cpukit/include/rtems/score/rbtree.h:523
523       while ( *link != NULL ) {
(gdb) where
#0  0x00008d54 in _RBTree_Insert_inline (less=<optimized out>, key=<synthetic pointer>, the_node=0x5ccc3d04, the_rbtree=0x736f4d74) at ../../../cpukit/include/rtems/score/rbtree.h:523
#1  _Priority_Plain_insert (priority=0, node=0x5ccc3d04, aggregation=0xc031 <vesa_realmode_bootup_init+3305>) at ../../../cpukit/include/rtems/score/priorityimpl.h:417
#2  _Thread_queue_Priority_do_enqueue (heads=0xc031 <vesa_realmode_bootup_init+3305>, queue_context=0x6c6c6544, the_thread=0x74 <_establish_stack+19>, queue=0x726f5020)
    at ../../../cpukit/score/src/threadqops.c:484
#3  _Thread_queue_Queue_enqueue (enqueue=<optimized out>, initialize=<optimized out>, queue_context=0x6c6c6544, the_thread=0x74 <_establish_stack+19>, queue=0x726f5020)
    at ../../../cpukit/score/src/threadqops.c:96
#4  _Thread_queue_Priority_enqueue (queue=0x726f5020, the_thread=0x74 <_establish_stack+19>, queue_context=0x6c6c6544) at ../../../cpukit/score/src/threadqops.c:566
#5  0x6c616972 in ?? ()
#6  0x00000000 in ?? ()
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00008d48 in _Chain_Insert_unprotected (the_node=0x283d4 <idtHdl+2708>, after_node=0x9002d0ff) at ../../../cpukit/include/rtems/score/chainimpl.h:666
666       before_node->previous = the_node;
(gdb) where
#0  0x00008d48 in _Chain_Insert_unprotected (the_node=0x283d4 <idtHdl+2708>, after_node=0x9002d0ff) at ../../../cpukit/include/rtems/score/chainimpl.h:666
#1  _Chain_Prepend_unprotected (the_node=0x283d4 <idtHdl+2708>, the_chain=0x9002d0ff) at ../../../cpukit/include/rtems/score/chainimpl.h:737
#2  _Thread_queue_Queue_enqueue (enqueue=<optimized out>, initialize=<optimized out>, queue_context=0x6c6c6544, the_thread=0x74 <_establish_stack+19>, queue=0x726f5020)
    at ../../../cpukit/score/src/threadqops.c:95
#3  _Thread_queue_Priority_enqueue (queue=0x726f5020, the_thread=0x74 <_establish_stack+19>, queue_context=0x6c6c6544) at ../../../cpukit/score/src/threadqops.c:566
#4  0x6c616972 in ?? ()
#5  0x00000000 in ?? ()
(gdb) c
Continuing.
^C
Program received signal SIGINT, Interrupt.
0x00008d54 in _RBTree_Insert_inline (less=<optimized out>, key=<synthetic pointer>, the_node=0x5ccc3d04, the_rbtree=0x736f4d74) at ../../../cpukit/include/rtems/score/rbtree.h:523
523       while ( *link != NULL ) {
(gdb) where
#0  0x00008d54 in _RBTree_Insert_inline (less=<optimized out>, key=<synthetic pointer>, the_node=0x5ccc3d04, the_rbtree=0x736f4d74) at ../../../cpukit/include/rtems/score/rbtree.h:523
#1  _Priority_Plain_insert (priority=0, node=0x5ccc3d04, aggregation=0xc031 <vesa_realmode_bootup_init+3305>) at ../../../cpukit/include/rtems/score/priorityimpl.h:417
#2  _Thread_queue_Priority_do_enqueue (heads=0xc031 <vesa_realmode_bootup_init+3305>, queue_context=0x6c6c6544, the_thread=0x74 <_establish_stack+19>, queue=0x726f5020)
    at ../../../cpukit/score/src/threadqops.c:484
#3  _Thread_queue_Queue_enqueue (enqueue=<optimized out>, initialize=<optimized out>, queue_context=0x6c6c6544, the_thread=0x74 <_establish_stack+19>, queue=0x726f5020)
    at ../../../cpukit/score/src/threadqops.c:96
#4  _Thread_queue_Priority_enqueue (queue=0x726f5020, the_thread=0x74 <_establish_stack+19>, queue_context=0x6c6c6544) at ../../../cpukit/score/src/threadqops.c:566
#5  0x6c616972 in ?? ()
#6  0x00000000 in ?? ()
(gdb) 

[Sebastian Huber]

Replying to kgardas:
[...]

Build builds both no-SMP and SMP BSPs fine, but neither of them runs on neither qemu nor on real hardware. If you find helpful, then qemu connected gdb shows those 3 traces over few seconds of BSP run:

With the old build system both configuration work I guess?

What is the qemu command line to test this BSP?

[kgardas]

Replying to Sebastian Huber:

Replying to kgardas:
[...]

Build builds both no-SMP and SMP BSPs fine, but neither of them runs on neither qemu nor on real hardware. If you find helpful, then qemu connected gdb shows those 3 traces over few seconds of BSP run:

With the old build system both configuration work I guess?

Yes, hello/smp works

What is the qemu command line to test this BSP?

qemu-system-i386 -m 128 -no-reboot -nographic -append "--video=off --console=/dev/com1" -kernel ./hello.exe

[Sebastian Huber <sebastian.huber@…>]

In 3be23809/rtems:

build: Fix i386/pc386 ABI flags

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In d9d31b3/rtems:

build: Add RELOCADDR to i386/pc386 options

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 8f424e45/rtems:

build: Fix i386/pc386 link of SMP start file

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 2a4ccc4/rtems:

bsp/pc386: Remove support for obsolete Binutils

Update #3818.

[Sebastian Huber]

Replying to kgardas:

Replying to Sebastian Huber:

Replying to kgardas:
[...]

Build builds both no-SMP and SMP BSPs fine, but neither of them runs on neither qemu nor on real hardware. If you find helpful, then qemu connected gdb shows those 3 traces over few seconds of BSP run:

With the old build system both configuration work I guess?

Yes, hello/smp works

With the recent fixes I was able to run the tests on Qemu.

What is the qemu command line to test this BSP?

qemu-system-i386 -m 128 -no-reboot -nographic -append "--video=off --console=/dev/com1" -kernel ./hello.exe

[kgardas]

Replying to Sebastian Huber:

With the recent fixes I was able to run the tests on Qemu.

Thanks a lot for fast fixes! I can confirm running no-SMP hello and SMP smp01 on both qemu and real hardware. Thanks!

[Sebastian Huber <sebastian.huber@…>]

In 845674e/rtems:

build: Fix mghttpd01 test exclude

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 11c1b862/rtems:

build: Allow test program item UIDs with a "-"

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 2b1a4ac/rtems:

build: Fix linker path

Use -B for the linker since some architectures use extra start files defined by
the GCC specification and provided by the BSP, for example powerpc. Start
files are not found by -L.

In the long run, the GCC defined start files should be provided by GCC. This
is a GCC 11 topic.

Update #3818

[Sebastian Huber <sebastian.huber@…>]

In d110e6e3/rtems:

bsp/gen83xx: Add missing source file

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 7392632/rtems:

build: Add RTEMS_PROFILING

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 2fb86e2/rtems-docs:

user: Add migration hints from old build system

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 9a27c9c/rtems:

build: Simplify get_compiler()

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In b8d8401/rtems:

build: Fix enabled-by evaluation

For the enabled-by evaluation we have to use the BSP name defined by the
build specification and not the user.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 71ccc15/rtems:

build: Fix option default values

For the option default values we have to use the BSP name defined by the build
specification and not the user.

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In cbd4f54/rtems:

build: Add comment

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 31c333d/rtems:

build: Include cpuopts.h in "config.h"

The Autoconf/Automake? build system did something similar. This fixes
the build of sigprogmask.c which uses RTEMS_POSIX_API and expects to get
it defined via "config.h".

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In f53712a/rtems:

build: Use exact match by default for BSP names

Update #3818.

[Sebastian Huber <sebastian.huber@…>]

In 94b8d85b/rtems:

build: Fix BSP name

Update #3818.

[Sebastian Huber]

The new build system is operational. Issues with the new build system should use new tickets.