#3516 closed enhancement (fixed)
sb-set-builder should report disk usage of build
Reported by: | Joel Sherrill | Owned by: | Chris Johns |
---|---|---|---|
Priority: | normal | Milestone: | 5.1 |
Component: | tool/rsb | Version: | 5 |
Severity: | normal | Keywords: | |
Cc: | Blocked By: | ||
Blocking: |
Description
Helping others work through the hello world, it is common for their VM images to not have enough disk space for the build to complete. It would be useful if the set-builder could report disk usage of the build/ directory. This information could be fed into the Users Guide.
It is frustrating and a bad experience to watch the build fail 90% of the way through.
Change History (10)
comment:1 Changed on 09/13/18 at 17:09:54 by Joel Sherrill
Owner: | set to Chris Johns |
---|---|
Status: | new → assigned |
comment:2 Changed on 09/13/18 at 18:28:23 by Joel Sherrill
comment:4 follow-up: 5 Changed on 09/15/18 at 23:38:54 by Chris Johns
What do you define as space that needs to be accounted for?
- The build tree after each package in a build set has been built?
- Installed files?
- Source and patches?
comment:5 Changed on 09/16/18 at 00:00:08 by Joel Sherrill
Replying to Chris Johns:
What do you define as space that needs to be accounted for?
- The build tree after each package in a build set has been built?
- Installed files?
- Source and patches?
Good questions. The ultimate answer is all of that. :)
The build tree peak is a critical piece of information. I saw people run out of disk space during the gcc/newlib phase. If we knew that, our instructions could give guidance. I did a no clean and was left with 8GB in build for SPARC. That's probably harsh if normally each package's build is cleaned. But SPARC now has more multilibs than I expected but arm and powerpc both have a lot also.
The question is "how much do I need to build the tools for architecture X?" That includes source tarballs, git, patches, peak space for untar'ed, and peak build usage.
And ideally we want to give advice that leaves people enough room to clone RTEMS and build it.
How can we get the input to give good advice that avoids folks running out of build space during the "getting started"?
comment:6 follow-up: 7 Changed on 09/16/18 at 03:35:55 by Chris Johns
I have taken a look at implementing this and it is not easy and complex to do so I do not think I will adding this feature.
The prep, build and clean are all part of a shell script that is generated and run and while I could add some du
commands to the script to dump the sizes there is no way to capture the output in a way that can be reported.
I feel it is simpler to create a script to wrap the sb-set-builder
command with the --no-clean
option and then dump the sizes once built. If this is run on a machine with plenty of storage you should be able to get the results to update the user manual.
comment:7 Changed on 09/17/18 at 04:06:24 by Chris Johns
Replying to Chris Johns:
I have taken a look at implementing this and it is not easy and complex to do so I do not think I will adding this feature.
The prep, build and clean are all part of a shell script that is generated and run and while I could add some
du
commands to the script to dump the sizes there is no way to capture the output in a way that can be reported.
I have taken a further look and it looks like the clean
phase of the script is not being used and the RSB is cleaning a build in Python. This makes implementing something possible. I still need to check all the .cfg
files to see if this is true.
The issue is any shell variables defined in one phase could be used in another phase and splitting the scripts means the variables or settings will not seen in subsequent phases.
comment:8 Changed on 09/20/18 at 03:20:07 by Chris Johns
I am concerned there is another issue which cause the overflow of a VM disk. What is in the 7.9G build
directory?
I have a patch I am testing which gives the following output for a rtems-sparc
build:
Build Sizes: usage: 3.661GB total: 2.412GB (sources: 1.472GB, patches: 6.630MB, installed 955.801MB)
The full output is:
$ ../source-builder/sb-set-builder --prefix=/opt/work/rtems/5 --log=5-sparc.txt 5/rtems-sparc RTEMS Source Builder - Set Builder, 5 (a16bfe19effa modified) Build Set: 5/rtems-sparc Build Set: 5/rtems-autotools.bset Build Set: 5/rtems-autotools-internal.bset config: tools/rtems-autoconf-2.69-1.cfg package: autoconf-2.69-x86_64-freebsd11.1-1 building: autoconf-2.69-x86_64-freebsd11.1-1 sizes: autoconf-2.69-x86_64-freebsd11.1-1: 7.506MB (installed: 0.000B) cleaning: autoconf-2.69-x86_64-freebsd11.1-1 config: tools/rtems-automake-1.12.6-1.cfg package: automake-1.12.6-x86_64-freebsd11.1-1 building: automake-1.12.6-x86_64-freebsd11.1-1 sizes: automake-1.12.6-x86_64-freebsd11.1-1: 8.326MB (installed: 0.000B) cleaning: automake-1.12.6-x86_64-freebsd11.1-1 cleaning: autoconf-2.69-x86_64-freebsd11.1-1 cleaning: automake-1.12.6-x86_64-freebsd11.1-1 Build Sizes: usage: 8.326MB total: 1.479GB (sources: 1.472GB, patches: 6.630MB, installed 0.000B) Build Set: Time 0:00:05.622352 Build Set: 5/rtems-autotools-base.bset config: tools/rtems-autoconf-2.69-1.cfg package: autoconf-2.69-x86_64-freebsd11.1-1 building: autoconf-2.69-x86_64-freebsd11.1-1 sizes: autoconf-2.69-x86_64-freebsd11.1-1: 7.503MB (installed: 3.033MB) cleaning: autoconf-2.69-x86_64-freebsd11.1-1 reporting: tools/rtems-autoconf-2.69-1.cfg -> autoconf-2.69-x86_64-freebsd11.1-1.txt reporting: tools/rtems-autoconf-2.69-1.cfg -> autoconf-2.69-x86_64-freebsd11.1-1.xml config: tools/rtems-automake-1.12.6-1.cfg package: automake-1.12.6-x86_64-freebsd11.1-1 building: automake-1.12.6-x86_64-freebsd11.1-1 sizes: automake-1.12.6-x86_64-freebsd11.1-1: 8.325MB (installed: 2.264MB) cleaning: automake-1.12.6-x86_64-freebsd11.1-1 reporting: tools/rtems-automake-1.12.6-1.cfg -> automake-1.12.6-x86_64-freebsd11.1-1.txt reporting: tools/rtems-automake-1.12.6-1.cfg -> automake-1.12.6-x86_64-freebsd11.1-1.xml installing: autoconf-2.69-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 installing: automake-1.12.6-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 cleaning: autoconf-2.69-x86_64-freebsd11.1-1 cleaning: automake-1.12.6-x86_64-freebsd11.1-1 Build Sizes: usage: 8.325MB total: 1.484GB (sources: 1.472GB, patches: 6.630MB, installed 5.298MB) Build Set: Time 0:00:07.382470 Build Set: Time 0:00:13.008615 config: devel/expat-2.1.0-1.cfg package: expat-2.1.0-x86_64-freebsd11.1-1 building: expat-2.1.0-x86_64-freebsd11.1-1 sizes: expat-2.1.0-x86_64-freebsd11.1-1: 4.171MB (installed: 623.483KB) cleaning: expat-2.1.0-x86_64-freebsd11.1-1 reporting: devel/expat-2.1.0-1.cfg -> expat-2.1.0-x86_64-freebsd11.1-1.txt reporting: devel/expat-2.1.0-1.cfg -> expat-2.1.0-x86_64-freebsd11.1-1.xml config: tools/rtems-binutils-2.31.1.cfg package: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 building: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 sizes: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1: 250.532MB (installed: 29.078MB) cleaning: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 reporting: tools/rtems-binutils-2.31.1.cfg -> sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1.txt reporting: tools/rtems-binutils-2.31.1.cfg -> sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1.xml config: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg package: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 building: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 sizes: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1: 3.661GB (installed: 846.775MB) cleaning: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 reporting: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg -> sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1.txt reporting: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg -> sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1.xml config: tools/rtems-gdb-8.0.1-1.cfg package: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 building: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 sizes: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1: 223.999MB (installed: 12.948MB) cleaning: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 reporting: tools/rtems-gdb-8.0.1-1.cfg -> sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1.txt reporting: tools/rtems-gdb-8.0.1-1.cfg -> sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1.xml config: tools/rtems-tools-5-1.cfg package: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 building: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 sizes: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1: 189.996MB (installed: 66.392MB) cleaning: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 reporting: tools/rtems-tools-5-1.cfg -> rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1.txt reporting: tools/rtems-tools-5-1.cfg -> rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1.xml config: tools/rtems-kernel-5.cfg package: sparc-rtems5-kernel-5-1 building: sparc-rtems5-kernel-5-1 sizes: sparc-rtems5-kernel-5-1: 8.308KB (installed: 0.000B) cleaning: sparc-rtems5-kernel-5-1 reporting: tools/rtems-kernel-5.cfg -> sparc-rtems5-kernel-5-1.txt reporting: tools/rtems-kernel-5.cfg -> sparc-rtems5-kernel-5-1.xml installing: expat-2.1.0-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 installing: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 installing: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 installing: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 -> /opt/work/rtems/5 installing: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 -> /opt/work/rtems/5 installing: sparc-rtems5-kernel-5-1 -> /opt/work/rtems/5 cleaning: expat-2.1.0-x86_64-freebsd11.1-1 cleaning: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 cleaning: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 cleaning: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 cleaning: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 cleaning: sparc-rtems5-kernel-5-1 Build Sizes: usage: 3.661GB total: 2.412GB (sources: 1.472GB, patches: 6.630MB, installed 955.801MB) Build Set: Time 0:20:56.188572
comment:9 Changed on 09/27/18 at 21:37:41 by Chris Johns <chrisj@…>
Resolution: | → fixed |
---|---|
Status: | assigned → closed |
I did a sparc tools build and it used 7.9 GB in build/. That probably means you need 12-15GB to have all the sources, rtems, and build RTEMS. Needs to be verified by others and to other architectures.