Add GCC, Binutils+GDB, Newlib mirrors to RTEMS github

[Sebastian Huber]

Sometimes it is necessary to use a snapshot of GCC, Binutils+GDB or Newlib. The snapshot archives of GCC and Binutils+GDB are only available for a certain time (e.g. half a year).

​https://lists.rtems.org/pipermail/devel/2018-June/021907.html

The sourceware.org Git services do not provide snapshots.

One option would be to use github since it provides snapshots. However, no official mirrors at github exist.

Add mirrors for GCC, Binutils+GDB and Newlib to ​https://github.com/rtems.

[Sebastian Huber]

My first step is to add

• ​https://github.com/rtems/binutils-gdb
• ​https://github.com/rtems/gcc
• ​https://github.com/rtems/newlib-cygwin

repositories, starting with the master branches only.

The second step is to add a cron job to dispatch.rtems.org to synchronize the repositories once per day with the upstream repositories.

Do we want to mirror all branches of the upstream repositories?

[Chris Johns]

What is practical?

The servers are on a fast backbone and so I do not see a big issue here.

[Sebastian Huber]

The least maintenance overhead has probably an automated mirroring of all branches since there are standard Git commands to do this.

[Sebastian Huber]

How are is the Git mirroring done for the RTEMS repositories? Can we use the same Git user account for this?

[Gedare Bloom]

This account ​https://github.com/rtems-updater
is used from dispatch to push commits in a git hook.
I don't know who has the rtems-updater credentials,
probably Amar set it up. I would guess its private key
should be on dispatch though.

[Chris Johns]

Can the repos please be under something like 'misc' or 'gnu' or 'fsf' and not at the top level.

I am concerned being at the top level will imply this is what users need to use.

Thanks

[Sebastian Huber]

I added the repositories.

I don't know you can add subdirectories in Github. Is this possible at all?

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

[Sebastian Huber]

The git clone --mirror checkout of the three repository need 2.5G of space on my disk. On dispatch.rtems.org we have currently:

df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/root0 88G 80G 1.4G 98% /

[Chris Johns]

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

[Chris Johns]

Replying to Sebastian Huber:

The git clone --mirror checkout of the three repository need 2.5G of space on my disk. On dispatch.rtems.org we have currently:

df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/root0 88G 80G 1.4G 98% /

What if we use the NAS under /data/ftp?

[Sebastian Huber]

Replying to Chris Johns:

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

I added pinned repositories.

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

What about gnu-mirror-gcc, sourceware-mirror-newlib-cygwin, and sourceware-mirror-binutils-gdb? This reflects the URL of the upstream Git repositories somehow.

[Chris Johns]

Replying to Sebastian Huber:

Replying to Chris Johns:

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

I added pinned repositories.

Thanks. Looks good.

I am wondering if I change the "Kernel, networking, file-systems, drivers, BSPs, samples, and testsuite." description on rtems to "Kernel (SMP), file-systems, drivers, BSPs, samples, and testsuite." and libbsd to "RTEMS FreeBSD, networking, USB, and media." ?

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

What about gnu-mirror-gcc, sourceware-mirror-newlib-cygwin, and sourceware-mirror-binutils-gdb? This reflects the URL of the upstream Git repositories somehow.

Nice. That makes sense to me.

[Sebastian Huber]

I have now the following setup on a local machine:

git clone --mirror git://sourceware.org/git/binutils-gdb.git
cd binutils-gdb
git remote add github git@github.com:RTEMS/sourceware-mirror-binutils-gdb.git

git clone --mirror git://sourceware.org/git/newlib-cygwin.git
cd newlib-cygwin
git remote add github git@github.com:RTEMS/sourceware-mirror-newlib-cygwin.git

git clone --mirror git://gcc.gnu.org/git/gcc.git/git/binutils-gdb.git
cd gcc
git remote add github git@github.com:RTEMS/gnu-mirror-gcc.git

Synchronization is quite easy:

git fetch
git push --mirror github

I will set up a cron job on my local machine for now. It runs during work days. To set it up on dispatch.rtems.org we need more disk space (I suggest 4G) and someone with enough privileges on this server.

[Chris Johns]

Replying to Sebastian Huber:

I will set up a cron job on my local machine for now. It runs during work days. To set it up on dispatch.rtems.org we need more disk space (I suggest 4G) and someone with enough privileges on this server.

Your local ftp directory under your personal area is on the NAS and should have enough space. Note, the NAS's FTP may not appear in the du list.

You can chmod the work space to stop it being scanned by search bots.

[Sebastian Huber]

I have at least two problems to set this up on dispatch.rtems.org:

1. I cannot add a user cron job:

   dispatch:~ # crontab -u sebh -e
   crontab: must be privileged to use -u
   dispatch:~ # sudo crontab -u sebh -e
   -bash: sudo: command not found
   

2. The Git mirror script forwards the request to a local server on port 8003 (/data/support/git-support/hooks/github-cmd-send.py). This server probably needs to have access to the local Git clone. I cannot set this up.

[Joel Sherrill]

Doesn't this work now? Sebastian/Chris? would you close the ticket or clarify what remains to do?