Python script distribution standardisation

[Chris Johns]

The RSB and Tools wrap python scripts in a shell script based wrapper that detects the installed version and uses it. Other python scripts in RTEMS do not. I would like to have a consistent model for all scripts.

​ PEP0394 discusses the issues and says a distro can decide not to provide a python command by default. It also says a virtualenv can be used to provide a consistent default environment.

Do we assume #! /usr/bin/env python will always work in our script and we make sure we are python2 and python3 clean for user installed commands?

If a user does not have a python command installed by default do we document how to install one or do we document using a virtualenv?

A simple shebang is cleaner for us to maintain however it added an extra dependency we need to document.

Note, development script and tools should follow the same model we decide on however they only need to support python3.

[Christian Mauderer]

I'm currently a bit undecided. Both methods have advantages and disadvantages.

Points that could be a reason _not_ to use a wrapper:

• One important point: We start to use waf everywhere. waf starts with a #!/usr/bin/env python. So it assumes that python exists. That means as soon as a user wants to build something with waf he needs a python. I don't think that we want to provide a wrapper for waf?
• Although you did a great job making it as simple as possible: A wrapper adds complexity. How many edge cases are missed in the wrappers? For example: If you try to call one of the source builder commands using a soft link in another directory, the $(dirname $0) of the wrapper will fail.
• Every supported system has to provide a /bin/sh and a dirname command. Can we be sure that's the case? What about Windows? If we enforce the wrapper we will always need a mingw or msys for Windows. If we remove it it might be possible to build RTEMS with only a python installation somewhen in the future (although you still need a cross-compiled gcc or llvm).
• Do we really want to support multiple python versions on the long term? It might start to get really complicated to test all relevant configurations. By the way: Does buildbot use python2, python3 or both?

Points that could be a good reason for using a wrapper:

• Do all our scripts work with python3 already? The wrapper of the source builder currently defaults to python2. So most likely it is nearly untested with python3.
• I think I had trouble building rtems-docs with python3 in the past (and I always have been too lazy to fix them - sorry). That means that currently _not_ everything is ready to work with all python versions. A wrapper could select the best one. But that also means we basically depend on python2 _and_ python3 to be installed.
• We will have a really simple possibility to switch to python4 when it comes out.
• On systems without a python it will definitively make it simpler.

Background questions:

• Do we know of distributions that don't provide a python (without version number) command?

[Sebastian Huber]

openSUSE 15.2 installs Python 3 by default and Python 2 is not available by default. There is no "python" command by default, only "python3".

I think we should ask the waf community how they want to deal with this.

About a year ago I started a discussion about "Which Python version for new tool code?". It seems RHEL 7 was the only relevant distribution not providing a Python 3, but this changed during the discussion:

​https://lists.rtems.org/pipermail/devel/2019-August/055010.html

CentOS 7.7 ships it also. Is there anything left which doesn't ship Python 3? If not, then why don't we change the shebang to #!/usr/bin/env python3?

[Chris Johns]

We need to support long life versions of host systems and I have understood this to mean python2. As a result we need to make sure our user facing tools run on Python 2 and 3 for a while yet. This ticket is about how we handle this and not that choice.

The python command may not install by default but is there a package that does provide it?

The waf point is a valid one. We need a python command while it requires it. If that changes then we can consider what happens then. If you wish to ask please do so.

[Christian Mauderer]

Seems that I misinterpreted the intention of the ticket. You asked:

Do we assume #! /usr/bin/env python will always work in our script and we make sure we are python2 and python3 clean for user installed commands?

If a user does not have a python command installed by default do we document how to install one or do we document using a virtualenv?

A simple shebang is cleaner for us to maintain however it added an extra dependency we need to document.

From my point of view: As long as waf needs python, we can assume it for our scripts too. So we have to document that anyway as soon as we switch to waf for all core repos.

From what you wrote we have to make sure that our scripts work with python2 and python3 anyway. I would reject a python2 only version with it beeing no maintained anymore and slowly removed from distributions and you don't like a python3 only version.

The alternative would be to add a shell wrapper around waf too. But do we really want to maintain that with all edge cases? And again: That adds a fixed dependency to a shell. Without a wrapper we might be able to have packages for Windows that only contain a binary toolchain and a python installation. That could be useful for big teams that want to build the toolchain only on one host and distribute it in binary form to all developers.

[Chris Johns]

I tend to agree. I would like to first know if CentOS7 and openSUSE 15.2 to have a package to install python?

[Sebastian Huber]

On the waf issue tracker I found this:

​https://gitlab.com/ita1024/waf/-/issues/2259

The suggested solution is to modify the shebang by hand for waf 2.*.

On openSUSE 15.2 you have to install Python 2 to get a python command or add the symlink yourself as root.

My preferred solution would be to change all shebangs to #!/usr/bin/env python3 and require all users of RTEMS to install Python 3.

[Chris Johns]

I have re-read PEP 394 and I am starting to think we should document and provide support for our users to run virtual environments. This is a recommendation in the PEP and I think it makes sense. As stated previously we need to support Python2 and Python3 for a period of time. Users restricted to Python2 should have a python command or they should have a virtualenv package for Python2. Python3 users should be on python3.6 or later and that has the venv module which gives them a python command.

I feel hard coding the version of python into the shebang restricts us and only creates further issues. Getting our users educated to use a virtual environment will pay off in the long term.

[Sebastian Huber]

I am not against using virtual environments, however, we should really evaluate if we want to support Python 2 in RTEMS 6. Maybe we should ask on the users mailing list if there are still host computer systems in use without a Python 3 support.

[Chris Johns]

A primary goal of RTEMS is to provide long life support for hardware and hosts.

I see the major issue with Python2 and Python3 as the shebang and python support rather than the coding. There are some basic things we need to do to support both but it is not a big deal. At this point in time I do not support just Python3.

As a result I prefer we focus on how we manage this situation. I believe virtual environments provide this. I see forcing the shebang to python3 as problematic long term.

[Sebastian Huber]

I still think we should try to learn a bit more about which hosts are in use by the RTEMS users. If there are still users around which are stuck with Python 2, then fine, otherwise we have one problem less.

How do you want to use the virtual environments? How is it set up? I have no experience in packaging and deploying Python projects. For me it would be a bit inconvenient if I have to activate a virtual environment just to run ./waf to build RTEMS for example.

[Joel Sherrill]

Can we write a simple program as an alternative to env which answers the question of "which Python" in a way we want? rtems-which-python python[23] and let it answer the question?

I am opposed to forcing use of virtualenv but only encouraging it. It is a nice feature and I am using it but I only use it because the host doesn't have everything in a complete (Tex packaging) or current way. If the host is sufficient, it shouldn't be needed.

CentOS 7 does have a Python 3 but you have to install a "software collection" (​https://developers.redhat.com/products/softwarecollections/overview for a description and list). This is the RedHat? way of adding newer technologies as an option. This is what I have been doing on CentOS 7 combined with a virtualenv.

scl enable rh-python36 bash

I have been doing the Python 3 Software Collection plus a virtualenv for Sphinx to build the RTEMS documentation.

[Christian Mauderer]

CentOS 7 does have a Python 3 but you have to install a "software collection" (​​https://developers.redhat.com/products/softwarecollections/overview for a description and list). This is the RedHat?? way of adding newer technologies as an option.

That's a bit odd. There is a python3 in the "os" repo of CentOS 7: ​http://mirror.centos.org/centos/7/os/x86_64/Packages/. That's the same repo where the kernel is (or at least a kernel). Is it possible that they added it?

[Chris Johns]

Using a virtual env, selecting an OS or a specific distribution of an OS, using python3 waf or adding a symlink yourself are all user choices. No one is forcing anyone to select any specific option and in fact I am concern we limit the choices in what we do. We also need to consider users may have other Python tools and commands and we need to make sure users can still run those tools.

The proposal I am putting forward is for user facing tools and scripts only and it is to shebang python only and we document a few of the approaches so it becomes a user choice. If you want to use CentOS with Python2 and python that is fine. If you want to install Python3 on CentOS that is no problem. If you use a bleeding edge Linux distro with a progressive set of packages and it only comes with Python3 that is also fine.

What I would like to avoid is us selecting a specific case and mandating it. This means we cannot force the shebang to python3 and we cannot explicitly exploit something in a virtual env.

Creating a specific env for us does not help other tools like waf.

[Christian Mauderer]

Maybe let me summarize a bit:

• No one really fought for keeping the wrappers. Instead we should start documenting how to set up a good environment (either via virtualenv or installing directly on host systems).

• For our tools we should select a shebang that is used in all user facing tools.

• We have at least waf which uses just python in the shebang. So regardless what we chose: Our environment needs a python command as long as waf uses it.

• There is no consensus about which python version we should use for user facing tools. Maybe we should move that discussion to the mailing list like suggested multiple times?

I think the first three points answer the original question of the ticket? Did I miss something from the discussion?

The last point is definitively a controversial one. There are good reasons for both directions. But again: Maybe that should be discussed with a broader audience?

[Chris Johns]

Replying to Christian Mauderer:

I think the first three points answer the original question of the ticket? Did I miss something from the discussion?

I agree and no I do not think you have.

The last point is definitively a controversial one. There are good reasons for both directions. But again: Maybe that should be discussed with a broader audience?

I think there will be some push back to dropping Python2 right now and this will fade as time goes on.

[Christian Mauderer]

The last point is definitively a controversial one. There are good reasons for both directions. But again: Maybe that should be discussed with a broader audience?

I think there will be some push back to dropping Python2 right now and this will fade as time goes on.

There would be some advantages too. Maybe we should discuss an indicator that it is time to drop the python2 support (for example: as soon as waf does; or as soon as CentOS doesn't installs it by default; or in 1 year; or ...) so that there is a clear plan and it isn't discussed again and again every few months. But like you said: That's not the topic of this ticket and I think it would be better to discuss this on the mailing list.

[Chris Johns]

Replying to Christian Mauderer:

The last point is definitively a controversial one. There are good reasons for both directions. But again: Maybe that should be discussed with a broader audience?

I think there will be some push back to dropping Python2 right now and this will fade as time goes on.

There would be some advantages too. Maybe we should discuss an indicator that it is time to drop the python2 support (for example: as soon as waf does; or as soon as CentOS doesn't installs it by default; or in 1 year; or ...) so that there is a clear plan and it isn't discussed again and again every few months. But like you said: That's not the topic of this ticket and I think it would be better to discuss this on the mailing list.

I think the new build system and the qual project need to be done and we have releases with them. I think at the core of this issue is releases with python 2 and 3 and once we know how they are traveling it will become apparent what we can do.

I will remove the script support in the RSB and close this ticket when it is merged.

[Chris Johns <chrisj@…>]

In b71f7f69/rtems-docs:

user, README: Add Python script host set up information

• Add Python3 and venv to the README

• Add a section on how to set up a host if the python command is not available.

Update #4037

[Chris Johns <chrisj@…>]

In 650c6f9/rtems-source-builder:

sb: Use shebang env python

Closes #4037

[Chris Johns]

The rtems-tools.git repo needs to be changed to a shebang env python.

[Chris Johns <chrisj@…>]

In c51f634/rtems-source-builder:

sb: Fix the imports on Python2

Updates #4037

[Chris Johns]

The user of a virtualenv is to be used to manage various versions of Python.