#3516 closed enhancement (fixed)

sb-set-builder should report disk usage of build

Reported by: Joel Sherrill Owned by: Chris Johns
Priority: normal Milestone: 5.1
Component: tool/rsb Version: 5
Severity: normal Keywords:
Cc: Blocked By:
Blocking:

Description

Helping others work through the hello world, it is common for their VM images to not have enough disk space for the build to complete. It would be useful if the set-builder could report disk usage of the build/ directory. This information could be fed into the Users Guide.

It is frustrating and a bad experience to watch the build fail 90% of the way through.

Change History (10)

comment:1 Changed on 09/13/18 at 17:09:54 by Joel Sherrill

Owner: set to Chris Johns
Status: newassigned

comment:2 Changed on 09/13/18 at 18:28:23 by Joel Sherrill

I did a sparc tools build and it used 7.9 GB in build/. That probably means you need 12-15GB to have all the sources, rtems, and build RTEMS. Needs to be verified by others and to other architectures.

comment:3 Changed on 09/13/18 at 23:38:24 by Chris Johns

Type: defectenhancement

comment:4 Changed on 09/15/18 at 23:38:54 by Chris Johns

What do you define as space that needs to be accounted for?

  1. The build tree after each package in a build set has been built?
  2. Installed files?
  3. Source and patches?

comment:5 in reply to:  4 Changed on 09/16/18 at 00:00:08 by Joel Sherrill

Replying to Chris Johns:

What do you define as space that needs to be accounted for?

  1. The build tree after each package in a build set has been built?
  2. Installed files?
  3. Source and patches?

Good questions. The ultimate answer is all of that. :)

The build tree peak is a critical piece of information. I saw people run out of disk space during the gcc/newlib phase. If we knew that, our instructions could give guidance. I did a no clean and was left with 8GB in build for SPARC. That's probably harsh if normally each package's build is cleaned. But SPARC now has more multilibs than I expected but arm and powerpc both have a lot also.

The question is "how much do I need to build the tools for architecture X?" That includes source tarballs, git, patches, peak space for untar'ed, and peak build usage.
And ideally we want to give advice that leaves people enough room to clone RTEMS and build it.

How can we get the input to give good advice that avoids folks running out of build space during the "getting started"?

comment:6 Changed on 09/16/18 at 03:35:55 by Chris Johns

I have taken a look at implementing this and it is not easy and complex to do so I do not think I will adding this feature.

The prep, build and clean are all part of a shell script that is generated and run and while I could add some du commands to the script to dump the sizes there is no way to capture the output in a way that can be reported.

I feel it is simpler to create a script to wrap the sb-set-builder command with the --no-clean option and then dump the sizes once built. If this is run on a machine with plenty of storage you should be able to get the results to update the user manual.

comment:7 in reply to:  6 Changed on 09/17/18 at 04:06:24 by Chris Johns

Replying to Chris Johns:

I have taken a look at implementing this and it is not easy and complex to do so I do not think I will adding this feature.

The prep, build and clean are all part of a shell script that is generated and run and while I could add some du commands to the script to dump the sizes there is no way to capture the output in a way that can be reported.

I have taken a further look and it looks like the clean phase of the script is not being used and the RSB is cleaning a build in Python. This makes implementing something possible. I still need to check all the .cfg files to see if this is true.

The issue is any shell variables defined in one phase could be used in another phase and splitting the scripts means the variables or settings will not seen in subsequent phases.

comment:8 Changed on 09/20/18 at 03:20:07 by Chris Johns

I am concerned there is another issue which cause the overflow of a VM disk. What is in the 7.9G build directory?

I have a patch I am testing which gives the following output for a rtems-sparc build:

Build Sizes: usage: 3.661GB total: 2.412GB (sources: 1.472GB, patches: 6.630MB, installed 955.801MB)

The full output is:

$ ../source-builder/sb-set-builder --prefix=/opt/work/rtems/5 --log=5-sparc.txt 5/rtems-sparc
RTEMS Source Builder - Set Builder, 5 (a16bfe19effa modified)
Build Set: 5/rtems-sparc
Build Set: 5/rtems-autotools.bset
Build Set: 5/rtems-autotools-internal.bset
config: tools/rtems-autoconf-2.69-1.cfg
package: autoconf-2.69-x86_64-freebsd11.1-1
building: autoconf-2.69-x86_64-freebsd11.1-1
sizes: autoconf-2.69-x86_64-freebsd11.1-1: 7.506MB (installed: 0.000B)
cleaning: autoconf-2.69-x86_64-freebsd11.1-1
config: tools/rtems-automake-1.12.6-1.cfg
package: automake-1.12.6-x86_64-freebsd11.1-1
building: automake-1.12.6-x86_64-freebsd11.1-1
sizes: automake-1.12.6-x86_64-freebsd11.1-1: 8.326MB (installed: 0.000B)
cleaning: automake-1.12.6-x86_64-freebsd11.1-1
cleaning: autoconf-2.69-x86_64-freebsd11.1-1
cleaning: automake-1.12.6-x86_64-freebsd11.1-1
Build Sizes: usage: 8.326MB total: 1.479GB (sources: 1.472GB, patches: 6.630MB, installed 0.000B)
Build Set: Time 0:00:05.622352
Build Set: 5/rtems-autotools-base.bset
config: tools/rtems-autoconf-2.69-1.cfg
package: autoconf-2.69-x86_64-freebsd11.1-1
building: autoconf-2.69-x86_64-freebsd11.1-1
sizes: autoconf-2.69-x86_64-freebsd11.1-1: 7.503MB (installed: 3.033MB)
cleaning: autoconf-2.69-x86_64-freebsd11.1-1
reporting: tools/rtems-autoconf-2.69-1.cfg -> autoconf-2.69-x86_64-freebsd11.1-1.txt
reporting: tools/rtems-autoconf-2.69-1.cfg -> autoconf-2.69-x86_64-freebsd11.1-1.xml
config: tools/rtems-automake-1.12.6-1.cfg
package: automake-1.12.6-x86_64-freebsd11.1-1
building: automake-1.12.6-x86_64-freebsd11.1-1
sizes: automake-1.12.6-x86_64-freebsd11.1-1: 8.325MB (installed: 2.264MB)
cleaning: automake-1.12.6-x86_64-freebsd11.1-1
reporting: tools/rtems-automake-1.12.6-1.cfg -> automake-1.12.6-x86_64-freebsd11.1-1.txt
reporting: tools/rtems-automake-1.12.6-1.cfg -> automake-1.12.6-x86_64-freebsd11.1-1.xml
installing: autoconf-2.69-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
installing: automake-1.12.6-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
cleaning: autoconf-2.69-x86_64-freebsd11.1-1
cleaning: automake-1.12.6-x86_64-freebsd11.1-1
Build Sizes: usage: 8.325MB total: 1.484GB (sources: 1.472GB, patches: 6.630MB, installed 5.298MB)
Build Set: Time 0:00:07.382470
Build Set: Time 0:00:13.008615
config: devel/expat-2.1.0-1.cfg
package: expat-2.1.0-x86_64-freebsd11.1-1
building: expat-2.1.0-x86_64-freebsd11.1-1
sizes: expat-2.1.0-x86_64-freebsd11.1-1: 4.171MB (installed: 623.483KB)
cleaning: expat-2.1.0-x86_64-freebsd11.1-1
reporting: devel/expat-2.1.0-1.cfg -> expat-2.1.0-x86_64-freebsd11.1-1.txt
reporting: devel/expat-2.1.0-1.cfg -> expat-2.1.0-x86_64-freebsd11.1-1.xml
config: tools/rtems-binutils-2.31.1.cfg
package: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1
building: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1
sizes: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1: 250.532MB (installed: 29.078MB)
cleaning: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1
reporting: tools/rtems-binutils-2.31.1.cfg -> sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1.txt
reporting: tools/rtems-binutils-2.31.1.cfg -> sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1.xml
config: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg
package: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1
building: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1
sizes: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1: 3.661GB (installed: 846.775MB)
cleaning: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1
reporting: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg -> sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1.txt
reporting: tools/rtems-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9.cfg -> sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1.xml
config: tools/rtems-gdb-8.0.1-1.cfg
package: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1
building: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1
sizes: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1: 223.999MB (installed: 12.948MB)
cleaning: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1
reporting: tools/rtems-gdb-8.0.1-1.cfg -> sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1.txt
reporting: tools/rtems-gdb-8.0.1-1.cfg -> sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1.xml
config: tools/rtems-tools-5-1.cfg
package: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1
building: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1
sizes: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1: 189.996MB (installed: 66.392MB)
cleaning: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1
reporting: tools/rtems-tools-5-1.cfg -> rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1.txt
reporting: tools/rtems-tools-5-1.cfg -> rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1.xml
config: tools/rtems-kernel-5.cfg
package: sparc-rtems5-kernel-5-1
building: sparc-rtems5-kernel-5-1
sizes: sparc-rtems5-kernel-5-1: 8.308KB (installed: 0.000B)
cleaning: sparc-rtems5-kernel-5-1
reporting: tools/rtems-kernel-5.cfg -> sparc-rtems5-kernel-5-1.txt
reporting: tools/rtems-kernel-5.cfg -> sparc-rtems5-kernel-5-1.xml
installing: expat-2.1.0-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
installing: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
installing: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
installing: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1 -> /opt/work/rtems/5
installing: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1 -> /opt/work/rtems/5
installing: sparc-rtems5-kernel-5-1 -> /opt/work/rtems/5
cleaning: expat-2.1.0-x86_64-freebsd11.1-1
cleaning: sparc-rtems5-binutils-2.31.1-x86_64-freebsd11.1-1
cleaning: sparc-rtems5-gcc-7.3.0-newlib-d13c84eb07e35984bf7a974cd786a6cdac29e6b9-x86_64-freebsd11.1-1
cleaning: sparc-rtems5-gdb-8.0.1-x86_64-freebsd11.1-1
cleaning: rtems-tools-d343f830f4dae8e84b4b44902347c60cf18b2ffd-1
cleaning: sparc-rtems5-kernel-5-1
Build Sizes: usage: 3.661GB total: 2.412GB (sources: 1.472GB, patches: 6.630MB, installed 955.801MB)
Build Set: Time 0:20:56.188572

comment:9 Changed on 09/27/18 at 21:37:41 by Chris Johns <chrisj@…>

Resolution: fixed
Status: assignedclosed

In 38fd56c/rtems-source-builder:

sb: Monitor the build disk usage. Report the usage, total and various sizes

  • Track the size of a build of a package in a build set to determine the maximum amout of disk space used. This can be used as a guide to documenting how much space a user needs to set aside to build a specific set of tools.
  • The %clean stage of a build is now split into a separate script. I do not think this is an issue because I could not find any %clean sections in any build configs we have. In time support for the %clean section will be removed, the package builder cleans up.

Closes #3516

comment:10 Changed on 09/27/18 at 22:18:50 by Chris Johns <chrisj@…>

In 079f95a/rtems-source-builder:

sb: Add build sizes to the email report.

  • Include build sizes in the email report.

Updates #3516

Note: See TracTickets for help on using tickets.