#3448 assigned task

Add GCC, Binutils+GDB, Newlib mirrors to RTEMS github

Reported by: Sebastian Huber Owned by: Sebastian Huber
Priority: normal Milestone: 5.1
Component: tool Version: 5
Severity: normal Keywords:
Cc: Blocked By:
Blocking:

Description

Sometimes it is necessary to use a snapshot of GCC, Binutils+GDB or Newlib. The snapshot archives of GCC and Binutils+GDB are only available for a certain time (e.g. half a year).

https://lists.rtems.org/pipermail/devel/2018-June/021907.html

The sourceware.org Git services do not provide snapshots.

One option would be to use github since it provides snapshots. However, no official mirrors at github exist.

Add mirrors for GCC, Binutils+GDB and Newlib to https://github.com/rtems.

Change History (15)

comment:1 Changed on Jun 11, 2018 at 8:31:19 AM by Sebastian Huber

Owner: changed from joel to Sebastian Huber

My first step is to add

repositories, starting with the master branches only.

The second step is to add a cron job to dispatch.rtems.org to synchronize the repositories once per day with the upstream repositories.

Do we want to mirror all branches of the upstream repositories?

comment:2 Changed on Jun 11, 2018 at 9:29:49 AM by Chris Johns

What is practical?

The servers are on a fast backbone and so I do not see a big issue here.

comment:3 Changed on Jun 11, 2018 at 9:51:22 AM by Sebastian Huber

The least maintenance overhead has probably an automated mirroring of all branches since there are standard Git commands to do this.

comment:4 Changed on Jun 11, 2018 at 9:56:25 AM by Sebastian Huber

How are is the Git mirroring done for the RTEMS repositories? Can we use the same Git user account for this?

comment:5 Changed on Jun 11, 2018 at 2:17:40 PM by Gedare

This account https://github.com/rtems-updater
is used from dispatch to push commits in a git hook.
I don't know who has the rtems-updater credentials,
probably Amar set it up. I would guess its private key
should be on dispatch though.

comment:6 Changed on Jun 12, 2018 at 1:48:22 AM by Chris Johns

Can the repos please be under something like 'misc' or 'gnu' or 'fsf' and not at the top level.

I am concerned being at the top level will imply this is what users need to use.

Thanks

comment:7 Changed on Jun 12, 2018 at 5:35:32 AM by Sebastian Huber

I added the repositories.

I don't know you can add subdirectories in Github. Is this possible at all?

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

comment:8 Changed on Jun 12, 2018 at 5:37:34 AM by Sebastian Huber

The git clone --mirror checkout of the three repository need 2.5G of space on my disk. On dispatch.rtems.org we have currently:

df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/root0 88G 80G 1.4G 98% /

comment:9 in reply to:  7 ; Changed on Jun 12, 2018 at 7:07:06 AM by Chris Johns

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

comment:10 in reply to:  8 Changed on Jun 12, 2018 at 7:08:08 AM by Chris Johns

Replying to Sebastian Huber:

The git clone --mirror checkout of the three repository need 2.5G of space on my disk. On dispatch.rtems.org we have currently:

df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/gpt/root0 88G 80G 1.4G 98% /

What if we use the NAS under /data/ftp?

comment:11 in reply to:  9 ; Changed on Jun 12, 2018 at 7:57:08 AM by Sebastian Huber

Replying to Chris Johns:

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

I added pinned repositories.

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

What about gnu-mirror-gcc, sourceware-mirror-newlib-cygwin, and sourceware-mirror-binutils-gdb? This reflects the URL of the upstream Git repositories somehow.

comment:12 in reply to:  11 Changed on Jun 12, 2018 at 10:33:51 PM by Chris Johns

Replying to Sebastian Huber:

Replying to Chris Johns:

Replying to Sebastian Huber:

I don't know you can add subdirectories in Github. Is this possible at all?

I thought you could but I must have not understood what I was looking at. I see there are pinned repos which might help us if we pin our ones so they are at the top.

I added pinned repositories.

Thanks. Looks good.

I am wondering if I change the "Kernel, networking, file-systems, drivers, BSPs, samples, and testsuite." description on rtems to "Kernel (SMP), file-systems, drivers, BSPs, samples, and testsuite." and libbsd to "RTEMS FreeBSD, networking, USB, and media." ?

We could use some sort of prefix, e.g. mirror-gcc, mirror-binutils-gdb, mirror-newlib-cygwin.

That is a good idea. fsf-mirror-gcc?.

What about gnu-mirror-gcc, sourceware-mirror-newlib-cygwin, and sourceware-mirror-binutils-gdb? This reflects the URL of the upstream Git repositories somehow.

Nice. That makes sense to me.

comment:13 Changed on Jun 13, 2018 at 5:33:44 AM by Sebastian Huber

I have now the following setup on a local machine:

git clone --mirror git://sourceware.org/git/binutils-gdb.git
cd binutils-gdb
git remote add github git@github.com:RTEMS/sourceware-mirror-binutils-gdb.git
git clone --mirror git://sourceware.org/git/newlib-cygwin.git
cd newlib-cygwin
git remote add github git@github.com:RTEMS/sourceware-mirror-newlib-cygwin.git
git clone --mirror git://gcc.gnu.org/git/gcc.git/git/binutils-gdb.git
cd gcc
git remote add github git@github.com:RTEMS/gnu-mirror-gcc.git

Synchronization is quite easy:

git fetch
git push --mirror github

I will set up a cron job on my local machine for now. It runs during work days. To set it up on dispatch.rtems.org we need more disk space (I suggest 4G) and someone with enough privileges on this server.

comment:14 in reply to:  13 Changed on Jun 13, 2018 at 5:58:49 AM by Chris Johns

Replying to Sebastian Huber:

I will set up a cron job on my local machine for now. It runs during work days. To set it up on dispatch.rtems.org we need more disk space (I suggest 4G) and someone with enough privileges on this server.

Your local ftp directory under your personal area is on the NAS and should have enough space. Note, the NAS's FTP may not appear in the du list.

You can chmod the work space to stop it being scanned by search bots.

comment:15 Changed on Jun 18, 2018 at 6:38:44 AM by Sebastian Huber

I have at least two problems to set this up on dispatch.rtems.org:

  1. I cannot add a user cron job:
    dispatch:~ # crontab -u sebh -e
    crontab: must be privileged to use -u
    dispatch:~ # sudo crontab -u sebh -e
    -bash: sudo: command not found
    
  1. The Git mirror script forwards the request to a local server on port 8003 (/data/support/git-support/hooks/github-cmd-send.py). This server probably needs to have access to the local Git clone. I cannot set this up.
Last edited on Jun 18, 2018 at 6:38:57 AM by Sebastian Huber (previous) (diff)
Note: See TracTickets for help on using tickets.