From 6fec09dc23bef28e8b7336f1054e7c80d2b3fb74 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Wed, 25 Mar 2020 15:22:58 -0400 Subject: [PATCH 01/22] src/doc/en/developer/portability_testing.rst: New --- src/doc/en/developer/index.rst | 8 + src/doc/en/developer/portability_testing.rst | 426 +++++++++++++++++++ 2 files changed, 434 insertions(+) create mode 100644 src/doc/en/developer/portability_testing.rst diff --git a/src/doc/en/developer/index.rst b/src/doc/en/developer/index.rst index f0e027a4aa6..3b4e44ff4c5 100644 --- a/src/doc/en/developer/index.rst +++ b/src/doc/en/developer/index.rst @@ -126,6 +126,14 @@ Running Sage's tests doctesting +Testing on multiple platforms +----------------------------- + +.. toctree:: + :maxdepth: 3 + + portability_testing + Contributing to Manuals and Tutorials ------------------------------------- diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst new file mode 100644 index 00000000000..97a8f592479 --- /dev/null +++ b/src/doc/en/developer/portability_testing.rst @@ -0,0 +1,426 @@ +.. nodoctest + +.. highlight:: shell-session + +.. _chapter-portability_testing: + +============================= +Testing on multiple platforms +============================= + +Sage is intended to build and run on a variety of platforms, +including all major Linux distributions, as well as macOS, and +Windows (with Cygwin). + +There is considerable variation between these platforms. +To ensure that Sage continues to build correctly on users' +machines, it is crucial to test changes to Sage, in particular +when external packages are added or upgraded, on a wide +spectrum of platforms. + +Sage patchbots +============== + +The Sage patchbots will automatically test your Trac ticket +by attempting an incremental build of Sage and running doctests. + +Sage buildbots +============== + +The Sage Release buildbot `_ +builds entire tarballs (e.g., all the development releases) on a +variety of machines. + +Developers' and users' tests on sage-release +============================================ + +Sage developers and users are encouraged to contribute to testing +releases that are announced on +https://groups.google.com/forum/#!forum/sage-release on their machines +and to report test results (success and failures) by responding to the +announcements. + +Testing Sage on a different platform using Docker +================================================= + +Docker (https://www.docker.com/) is a popular virtualization software, +running Linux operating system images in containers on a shared Linux +kernel. Using Docker Desktop for Mac and Windows, the containers can +also be run on these platforms. + +All major Linux distributions provide ready-to-use Docker images, +which are published at https://hub.docker.com. For example, to run +the current stable (LTS) version of Ubuntu interactively, you can use +the shell command:: + + [mkoeppe@sage sage]$ docker run -it ubuntu:latest + root@9f3398da43c2:/# + +Here ``ubuntu`` is referred to as the "image (name)" and ``latest`` as +the "tag". Other releases of Ubuntu are available under different +tags, such as ``xenial`` or ``devel``. + +The above command drops you in a root shell on the container:: + + root@9f3398da43c2:/# uname -a + Linux 9f3398da43c2 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux + root@9f3398da43c2:/# df -h + Filesystem Size Used Avail Use% Mounted on + overlay 181G 116G 56G 68% / + tmpfs 64M 0 64M 0% /dev + tmpfs 2.7G 0 2.7G 0% /sys/fs/cgroup + shm 64M 0 64M 0% /dev/shm + /dev/sda1 181G 116G 56G 68% /etc/hosts + tmpfs 2.7G 0 2.7G 0% /proc/acpi + tmpfs 2.7G 0 2.7G 0% /sys/firmware + +Exiting the shell terminates the container:: + + root@9f3398da43c2:/# ^D + [mkoeppe@sage sage]$ + +Let us work with a distclean Sage source tree. If you are using git, +a good way to get one (without losing a precious installation in +``SAGE_LOCAL``) is by creating a new worktree:: + + [mkoeppe@sage sage] git worktree add worktree-ubuntu-latest + [mkoeppe@sage sage] cd worktree-ubuntu-latest + [mkoeppe@sage worktree-ubuntu-latest] ls + COPYING.txt ... Makefile ... configure.ac ... src tox.ini + +This is not bootstrapped (``configure`` is missing), so let's bootstrap it:: + + [mkoeppe@sage worktree-ubuntu-latest] make configure + ... + +We can start a container again with same image, ``ubuntu:latest``, but this time let's +mount the current directory into it:: + + [mkoeppe@sage worktree-ubuntu-latest]$ docker run -it --mount type=bind,source=$(pwd),target=/sage ubuntu:latest + root@39d693b2a75d:/# mount | grep sage + osxfs on /sage type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576) + root@39d693b2a75d:/# cd sage + root@39d693b2a75d:/sage# ls + COPYING.txt ... Makefile ... config configure configure.ac ... src tox.ini + +Typical Docker images provide minimal installations of packages only:: + + root@39d693b2a75d:/sage# command -v python + root@39d693b2a75d:/sage# command -v gcc + root@39d693b2a75d:/sage# + +As you can see above, the image ``ubuntu:latest`` has neither a Python nor +a GCC installed, which are among the build prerequisites of Sage. We +need to install them using the distribution's package manager first. + +Sage facilitates testing various distributions on Docker as follows. + +Discovering the system's package system +--------------------------------------- + +:: + + root@39d693b2a75d:/sage# build/bin/sage-guess-package-system + debian + +Let's install gcc, hoping that the Ubuntu package providing it is +simply named ``gcc``. If we forgot what the package manager on +Debian-derived distributions is called, we can ask Sage for a +reminder:: + + root@39d693b2a75d:/sage# build/bin/sage-print-system-package-command debian install gcc + sudo apt-get install gcc + +We are already root, so we can drop the ``sudo``, of course:: + + root@39d693b2a75d:/sage# apt-get install gcc + +.. actually this needs an apt-get update first. + +Using Sage's database of distribution prerequisites +--------------------------------------------------- + +The source code of the Sage distribution contains a database of +package names in various distributions' package managers. For +example, the file ``build/pkgs/debian.txt`` contains the following:: + + # This file, build/pkgs/debian.txt, contains names of Debian/Ubuntu packages + # needed for installation of Sage from source. + # + # In addition, the files build/pkgs/SPKG/debian.txt contain the names + # of packages that provide the equivalent of SPKG. + # + # Everything on a line after a # character is ignored. + binutils + make + m4 + perl + # python3-minimal is not enough on debian buster, ubuntu bionic - it does not have urllib + python3 # system python for bootstrapping the build + tar + bc + gcc + # On debian buster, need C++ even to survive 'configure'. Otherwise: + # checking how to run the C++ preprocessor... /lib/cpp + # configure: error: in `/sage': + # configure: error: C++ preprocessor "/lib/cpp" fails sanity check + g++ + # Needed if we download some packages from a https upstream URL + ca-certificates + +From this information, we know that we can use the following command +on our container to install the necessary build prerequisites:: + + root@39d693b2a75d:/sage# apt-get install binutils make m4 perl python3 tar bc gcc g++ ca-certificates + Reading package lists... Done + Building dependency tree + Reading state information... Done + tar is already the newest version (1.29b-2ubuntu0.1). + The following additional packages will be installed: + ... + Done. + +Now we can start the build:: + + root@39d693b2a75d:/sage# ./configure + checking for a BSD-compatible install... /usr/bin/install -c + checking for root user... yes + configure: error: You cannot build Sage as root, switch to an unprivileged user. (If building in a container, use --enable-build-as-root.) + +Let's just follow this helpful hint:: + + root@39d693b2a75d:/sage# ./configure --enable-build-as-root + checking for a BSD-compatible install... /usr/bin/install -c + ... + + +Using Sage's database of equivalent distribution packages +--------------------------------------------------------- + +At the end of the ``configure`` run, Sage issued a message like the +following:: + + configure: notice: the following SPKGs did not find equivalent system packages: arb boost boost_cropped bzip2 ... yasm zeromq zlib + checking for the package system in use... debian + configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: + configure: $ sudo apt-get install libflint-arb-dev ... yasm libzmq3-dev libz-dev + configure: After installation, re-run configure using: + configure: $ ./config.status --recheck && ./config.status + +This information comes from Sage's database of equivalent distribution +packages. For example:: + + root@39d693b2a75d:/sage# ls build/pkgs/arb/distros/ + arch.txt conda.txt debian.txt gentoo.txt + root@39d693b2a75d:/sage# cat build/pkgs/arb/distros/debian.txt + libflint-arb-dev + +Note that these package equivalencies are based on a current stable or +testing version of the distribution; the packages are not guaranteed +to exist in every release or derivative distribution. + +The Sage distribution is intended to build correctly no matter what +superset of the set of packages forming the minimal build +prerequisites is installed on the system. If it does not, this is a +bug of the Sage distribution and should be reported and fixed on a +ticket. Crucial part of a bug report is the configuration of the +system, in particular a list of installed packages and their versions. + +Let us install a subset of these packages:: + + root@39d693b2a75d:/sage# apt-get install libbz2-dev bzip2 yasm libz-dev + Reading package lists... Done + ... + Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... + root@39d693b2a75d:/sage# + + +Committing a container to disk +------------------------------ + +After terminating the container, we can create a new image +corresponding to its current state:: + + root@39d693b2a75d:/sage# ^D + [mkoeppe@sage worktree-ubuntu-latest]$ docker ps -a | head -n2 + CONTAINER ID IMAGE COMMAND CREATED STATUS + 39d693b2a75d ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 6 seconds ago + 9f3398da43c2 ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 8 minutes ago + [mkoeppe@sage worktree-ubuntu-latest]$ docker commit 39d693b2a75d ubuntu-latest-minimal-17 + sha256:4151c5ca4476660f6181cdb13923da8fe44082222b984c377fb4fd6cc05415c1 + +Here, ``39d693b2a75d`` was the container id (which appeared in the +shell prompts and in the output of ``docker ps``), and +``ubuntu-latest-minimal-17`` is an arbitrary symbolic name for the new +image. The output of the command is the id of the new image. We can +use either the symbolic name or the id to refer to the new image. + +We can run the image and get a new container with the same state as +the one that we terminated. Again we want to mount our worktree into +it; otherwise, because we did not make a copy, the new container will +have no access to the worktree:: + + [mkoeppe@sage worktree-ubuntu-latest]$ docker run -it \ + --mount type=bind,source=$(pwd),target=/sage ubuntu-latest-minimal-17 + root@73987568712c:/# cd sage + root@73987568712c:/sage# command -v gcc + /usr/bin/gcc + root@73987568712c:/sage# command -v yasm + /usr/bin/yasm + root@73987568712c:/sage# ^D + [mkoeppe@sage worktree-ubuntu-latest]$ + +The image ``ubuntu-latest-minimal-17`` can be run in as many +containers as we want and can also be shared with other users or +developers so that they can run it in a container on their machine. +(See the Docker documentation on how to do this.) + +This facilitates collaboration on fixing portability bugs of the Sage +distribution. After reproducing a portability bug on a container, +several developers can work on fixing the bug using containers running +on their respective machines. + + +Generating Dockerfiles +---------------------- + +Sage also provides a script for generating a ``Dockerfile``, which is +a recipe for automatically building a new image:: + + [mkoeppe@sage sage]$ build/bin/write-dockerfile.sh debian "@(standard|optional)" > Dockerfile + +(The second argument is a bash extglob pattern for the package type.) + +.. this interface should be improved obviously. See #29146 - Refactor tox.ini and build/bin/write_dockerfile.sh + +The ``Dockerfile`` instructs the command ``docker build`` to start +from a base image, install system packages, and then bootstrap, +configure, build, and test Sage. + +In contrast to the workflow described in the above sections, the +``Dockerfile`` **copies** the Sage worktree into the build container, +using ``ADD`` commands. Because the copying is subject to the +exclusions in the ``.gitignore`` file (via a symbolic link from +``.dockerignore``), only the sources are copied, but none of the build +or installation artefacts. Therefore, it can be used without any +problems also for worktrees that are not distclean. + +You can customize the image build process by passing build arguments to the +command ``docker build``. For example:: + + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg BASE_IMAGE=ubuntu:latest \ + --build-arg NUMPROC=4 \ + --build-arg EXTRA_CONFIGURE_ARGS="--with-python=2" + +These arguments are defined using ``ARG`` commands in the +``Dockerfile``. + +You can customize the image build process further by editing the +``Dockerfile``. For example, by default, the generated ``Dockerfile`` +configures, builds, and tests Sage. By deleting or commenting out the +commands for the latter, you can adjust the Dockerfile to stop after +the ``configure`` phase. + + +Symbolic names of system configurations +--------------------------------------- + +To facilitate communication about host system configurations and for +automating testing, Sage defines symbolic names composed of several +"factors" +(https://tox.readthedocs.io/en/latest/config.html#complex-factor-conditions) +in the file ``$SAGE_ROOT/tox.ini``. + +The **system factor** describes a base operating system +image. + +- Examples are ``ubuntu-focal``, ``debian-buster``, + ``archlinux-latest``, ``fedora-30``, ``slackware-14.2``, + ``centos-7-i386``, and ``ubuntu-bionic-arm64``. + +- See ``$SAGE_ROOT/tox.ini`` for a complete list, and to which images + on Docker hub they correspond. + +The **packages factor** describes a list of system packages to be +installed on the system before building Sage: + +- ``minimal`` installs the system packages known to Sage to provide + minimal prerequisites for bootstrapping and building the Sage + distribution. + +- ``standard`` additionally installs all known system packages that + are equivalent to standard packages of the Sage distribution, for + which the mechanism ``spkg-configure.m4`` is implemented. + This corresponds to the type pattern ``@(standard)``. + +- ``maximal`` does the same for all standard and optional packages. + This corresponds to the type pattern ``@(standard|optional)``. + +The factors are connected by a hyphen to name a system configuration, +such as ``debian-buster-standard`` and ``centos-7-i386-minimal``. + + +Automatic Docker-based build testing using tox +---------------------------------------------- + +``tox`` is Python package that is widely used for automating tests of +Python projects. + +Install ``tox`` for use with your system Python, for example using:: + + [mkoeppe@sage sage]$ pip install --user tox + +A tox "environment" is identified by a name composed of several +factors. In addition to the system factor and the packages factor +that we introduced above, there are the following factors: + +The **technology** factor describes how the environment is run: + +- ``docker`` builds a Docker image as described above + +.. + - ``local`` runs testing on the host OS instead. This is currently only + implemented for ``local-homebrew-macos``; see below. + +The **configuration** factor (allowed to be empty): + +- ``python2`` adds the argument ``--with-python=2`` to the + ``configure`` run. + +The factors are connected by a hyphen to name a tox environment. To +run an environment:: + + [mkoeppe@sage sage]$ tox -e docker-slackware-14.2-minimal + [mkoeppe@sage sage]$ tox -e docker-ubuntu-bionic-standard-python2 + +Extra arguments to ``docker build`` can be supplied through the +environment variable ``EXTRA_DOCKER_BUILD_ARGS``. For example, for +a silent build (``make V=0``), use:: + + [mkoeppe@sage sage]$ EXTRA_DOCKER_BUILD_ARGS="--build-arg USE_MAKEFLAGS=\"V=0\"" \ + tox -e docker-ubuntu-bionic-standard + + +Automatic parallel tox runs on GitHub Actions +--------------------------------------------- + +The Sage source tree includes a default configuration for GitHub +Actions that runs tox on a multitude of platforms on every pull +request to a repository for which GitHub Actions are enabled. + +This is defined in the file ``$SAGE_ROOT/.github/workflows/tox.yml``. + +An additional GitHub Actions workflow for testing on Cygwin, not based +on tox, is defined in the file +``$SAGE_ROOT/.github/workflows/ci-cygwin.yml``. + + +.. + Workflow for responding to portability bug reports + ================================================== + + 1. Reproduce the platform and the bug. + + 2. Fix the bug. + From dc3d238125cfdb8aadfb26b65f26dd454a4ca509 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Wed, 25 Mar 2020 20:57:33 -0400 Subject: [PATCH 02/22] add info on GitHub Actions runners --- src/doc/en/developer/portability_testing.rst | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 97a8f592479..c5803f64506 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -415,7 +415,14 @@ An additional GitHub Actions workflow for testing on Cygwin, not based on tox, is defined in the file ``$SAGE_ROOT/.github/workflows/ci-cygwin.yml``. - +GitHub Actions runs these build jobs on 2-core machines with 7 GB of +RAM memory and 14 GB of SSD disk space +(https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources), +and has a time limit of 6h per job. This is just barely enough for a +typical ``minimal`` build followed by make ptest to succeed; and +plenty of time for a typical ``standard`` build to succeed. + + .. Workflow for responding to portability bug reports ================================================== From a476300fa9128c0c56dfc64243c03f65b599b7f4 Mon Sep 17 00:00:00 2001 From: Dima Pasechnik Date: Thu, 26 Mar 2020 11:18:46 -0400 Subject: [PATCH 03/22] Reviewer patch --- src/doc/en/developer/portability_testing.rst | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index c5803f64506..268503ce5fb 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -9,7 +9,7 @@ Testing on multiple platforms ============================= Sage is intended to build and run on a variety of platforms, -including all major Linux distributions, as well as macOS, and +including all major Linux distributions, as well as MacOS, and Windows (with Cygwin). There is considerable variation between these platforms. @@ -142,7 +142,9 @@ Using Sage's database of distribution prerequisites The source code of the Sage distribution contains a database of package names in various distributions' package managers. For -example, the file ``build/pkgs/debian.txt`` contains the following:: +example, the file ``build/pkgs/debian.txt`` contains the following + +.. code-block:: yaml # This file, build/pkgs/debian.txt, contains names of Debian/Ubuntu packages # needed for installation of Sage from source. From ffdbdc875501b508509a0afc8a2f6d12944f9f9c Mon Sep 17 00:00:00 2001 From: Dima Pasechnik Date: Thu, 26 Mar 2020 14:02:04 -0400 Subject: [PATCH 04/22] Reviewer patch --- src/doc/en/developer/portability_testing.rst | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 268503ce5fb..20737e3ce12 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -21,13 +21,13 @@ spectrum of platforms. Sage patchbots ============== -The Sage patchbots will automatically test your Trac ticket +The `Sage patchbots `_ will automatically test your Trac ticket by attempting an incremental build of Sage and running doctests. Sage buildbots ============== -The Sage Release buildbot `_ +The `Sage Release buildbot `_ builds entire tarballs (e.g., all the development releases) on a variety of machines. @@ -36,20 +36,20 @@ Developers' and users' tests on sage-release Sage developers and users are encouraged to contribute to testing releases that are announced on -https://groups.google.com/forum/#!forum/sage-release on their machines +`Sage Release `_ on their machines and to report test results (success and failures) by responding to the announcements. Testing Sage on a different platform using Docker ================================================= -Docker (https://www.docker.com/) is a popular virtualization software, +`Docker `_ is a popular virtualization software, running Linux operating system images in containers on a shared Linux kernel. Using Docker Desktop for Mac and Windows, the containers can also be run on these platforms. All major Linux distributions provide ready-to-use Docker images, -which are published at https://hub.docker.com. For example, to run +which are published via `Docker Hub `_. For example, to run the current stable (LTS) version of Ubuntu interactively, you can use the shell command:: @@ -199,7 +199,7 @@ Let's just follow this helpful hint:: Using Sage's database of equivalent distribution packages --------------------------------------------------------- -At the end of the ``configure`` run, Sage issued a message like the +At the end of the ``./configure`` run, Sage issued a message like the following:: configure: notice: the following SPKGs did not find equivalent system packages: arb boost boost_cropped bzip2 ... yasm zeromq zlib @@ -330,9 +330,7 @@ Symbolic names of system configurations To facilitate communication about host system configurations and for automating testing, Sage defines symbolic names composed of several -"factors" -(https://tox.readthedocs.io/en/latest/config.html#complex-factor-conditions) -in the file ``$SAGE_ROOT/tox.ini``. +`Tox "factors" `_ in the file ``$SAGE_ROOT/tox.ini``. The **system factor** describes a base operating system image. @@ -418,8 +416,8 @@ on tox, is defined in the file ``$SAGE_ROOT/.github/workflows/ci-cygwin.yml``. GitHub Actions runs these build jobs on 2-core machines with 7 GB of -RAM memory and 14 GB of SSD disk space -(https://help.github.com/en/actions/reference/virtual-environments-for-github-hosted-runners#supported-runners-and-hardware-resources), +RAM memory and 14 GB of SSD disk space, cf. +`here `_, and has a time limit of 6h per job. This is just barely enough for a typical ``minimal`` build followed by make ptest to succeed; and plenty of time for a typical ``standard`` build to succeed. From af6e904a998789ef1d91ceb905db99d89f41a11e Mon Sep 17 00:00:00 2001 From: "John H. Palmieri" Date: Thu, 26 Mar 2020 17:27:43 -0400 Subject: [PATCH 05/22] Reviewer patch --- src/doc/en/developer/portability_testing.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 20737e3ce12..00abe9931ba 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -12,7 +12,7 @@ Sage is intended to build and run on a variety of platforms, including all major Linux distributions, as well as MacOS, and Windows (with Cygwin). -There is considerable variation between these platforms. +There is considerable variation among these platforms. To ensure that Sage continues to build correctly on users' machines, it is crucial to test changes to Sage, in particular when external packages are added or upgraded, on a wide From ae6e20fc833598b3652a6af57a29f3aad978559e Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 17:29:44 -0400 Subject: [PATCH 06/22] src/doc/en/developer/portability_testing.rst: Mention apt-get update --- src/doc/en/developer/portability_testing.rst | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 00abe9931ba..81373f7965f 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -131,12 +131,13 @@ reminder:: root@39d693b2a75d:/sage# build/bin/sage-print-system-package-command debian install gcc sudo apt-get install gcc -We are already root, so we can drop the ``sudo``, of course:: +We are already root, so we can drop the ``sudo``, of course. +And we remember that we need to fetch the current package lists +from the server first:: + root@39d693b2a75d:/sage# apt-get update root@39d693b2a75d:/sage# apt-get install gcc -.. actually this needs an apt-get update first. - Using Sage's database of distribution prerequisites --------------------------------------------------- From 8d876b07effbe37f72738ed3a9560779f3311d2e Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 19:03:06 -0400 Subject: [PATCH 07/22] Add more details --- src/doc/en/developer/portability_testing.rst | 257 ++++++++++++++++++- 1 file changed, 243 insertions(+), 14 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 81373f7965f..1135ff43cdb 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -245,7 +245,7 @@ After terminating the container, we can create a new image corresponding to its current state:: root@39d693b2a75d:/sage# ^D - [mkoeppe@sage worktree-ubuntu-latest]$ docker ps -a | head -n2 + [mkoeppe@sage worktree-ubuntu-latest]$ docker ps -a | head -n3 CONTAINER ID IMAGE COMMAND CREATED STATUS 39d693b2a75d ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 6 seconds ago 9f3398da43c2 ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 8 minutes ago @@ -296,17 +296,55 @@ a recipe for automatically building a new image:: .. this interface should be improved obviously. See #29146 - Refactor tox.ini and build/bin/write_dockerfile.sh -The ``Dockerfile`` instructs the command ``docker build`` to start -from a base image, install system packages, and then bootstrap, -configure, build, and test Sage. +The ``Dockerfile`` instructs the command ``docker build`` to build a +new Docker image. Let us take a quick look at the generated file; +this is slightly simplified:: -In contrast to the workflow described in the above sections, the -``Dockerfile`` **copies** the Sage worktree into the build container, -using ``ADD`` commands. Because the copying is subject to the -exclusions in the ``.gitignore`` file (via a symbolic link from -``.dockerignore``), only the sources are copied, but none of the build -or installation artefacts. Therefore, it can be used without any -problems also for worktrees that are not distclean. + [mkoeppe@sage sage]$ cat Dockerfile + # Automatically generated by SAGE_ROOT/build/bin/write-dockerfile.sh + # the :comments: separate the generated file into sections + # to simplify writing scripts that customize this file + ... + +First, it instructs ``docker build`` to start from an existing base +image...:: + + ... + ARG BASE_IMAGE=ubuntu:latest + FROM ${BASE_IMAGE} + ... + +Then, to install system packages...:: + + ... + RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes binutils make m4 perl python3 ... yasm libzmq3-dev libz-dev && apt-get clean + +Then, to bootstrap and configure...:: + + RUN mkdir -p /sage + WORKDIR /sage + ADD Makefile VERSION.txt README.md bootstrap configure.ac sage ./ + ADD src/doc/bootstrap src/doc/bootstrap + ADD m4 ./m4 + ADD build ./build + ADD src/bin/sage-version.sh src/bin/sage-version.sh + RUN ./bootstrap + ADD src/bin src/bin + ADD src/Makefile.in src/Makefile.in + ARG EXTRA_CONFIGURE_ARGS="" + RUN ./configure --enable-build-as-root --enable-option-checking ${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1) + +Finally, to build and test...:: + + ARG NUMPROC=8 + ENV MAKE="make -j${NUMPROC}" + ARG USE_MAKEFLAGS="-k" + RUN make ${USE_MAKEFLAGS} base-toolchain + ARG TARGETS_PRE="sagelib-build-deps" + RUN make ${USE_MAKEFLAGS} ${TARGETS_PRE} + ADD src src + ARG TARGETS="build ptest" + RUN make ${USE_MAKEFLAGS} ${TARGETS} You can customize the image build process by passing build arguments to the command ``docker build``. For example:: @@ -316,15 +354,206 @@ command ``docker build``. For example:: --build-arg NUMPROC=4 \ --build-arg EXTRA_CONFIGURE_ARGS="--with-python=2" -These arguments are defined using ``ARG`` commands in the -``Dockerfile``. +These arguments (and their default values) are defined using ``ARG`` +commands in the ``Dockerfile``. + +The above command will build Sage from scratch and will therefore take +quite long. Let us instead just do a partial build, consisting of one +small package, by setting the arguments ``TARGETS_PRE`` and +``TARGETS``. We use a silent build (``make V=0``):: + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg TARGETS_PRE=ratpoints \ + --build-arg TARGETS=ratpoints \ + --build-arg USE_MAKEFLAGS="V=0" + Sending build context to Docker daemon 285MB + Step 1/28 : ARG BASE_IMAGE=ubuntu:latest + ... + Step 2/28 : FROM ${BASE_IMAGE} + ---> 549b9b86cb8d + ... + Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE} + ... + make[1]: Entering directory '/sage/build/make' + sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '/sage/logs/pkgs/ratpoints-2.1.3.p5.log' + [ratpoints-2.1.3.p5] installing. Log file: /sage/logs/pkgs/ratpoints-2.1.3.p5.log + [ratpoints-2.1.3.p5] successfully installed. + make[1]: Leaving directory '/sage/build/make' + + real 0m18.886s + user 0m1.779s + sys 0m0.314s + Sage build/upgrade complete! + ... + ---> 2d06689d39fa + Successfully built 2d06689d39fa + +We can now start a container using the image id shown in the last step:: + + [mkoeppe@sage sage]$ docker run -it 2d06689d39fa bash + root@fab59e09a641:/sage# ls -l logs/pkgs/ + total 236 + -rw-r--r-- 1 root root 231169 Mar 26 22:07 config.log + -rw-r--r-- 1 root root 6025 Mar 26 22:27 ratpoints-2.1.3.p5.log + root@fab59e09a641:/sage# ls -l local/lib/*rat* + -rw-r--r-- 1 root root 177256 Mar 26 22:27 local/lib/libratpoints.a + You can customize the image build process further by editing the ``Dockerfile``. For example, by default, the generated ``Dockerfile`` configures, builds, and tests Sage. By deleting or commenting out the commands for the latter, you can adjust the Dockerfile to stop after -the ``configure`` phase. +the ``configure`` phase, for example. + +``Dockerfile`` is the default filename for Dockerfiles. You can +change it to any other name, but it is recommended to use +``Dockerfile`` as a prefix, such as ``Dockerfile-debian-standard``. +It should be placed within the tree rooted at the current directory +(``.``); if you want to put it elsewhere, you need to learn about +details of "Docker build contexts". + +Note that in contrast to the workflow described in the above sections, +the ``Dockerfile`` **copies** a snapshot of your Sage worktree into +the build container, using ``ADD`` commands, instead of mounting the +directory into it. This copying is subject to the exclusions in the +``.gitignore`` file (via a symbolic link from ``.dockerignore``). +Therefore, only the sources are copied, but not your configuration +(such as the file ``config.status``), nor the ``$SAGE_LOCAL`` tree, +nor any other build artefacts. + +Because of this, you can build a Docker image using the generated +``Dockerfile`` from your main Sage development tree. It does not have +to be distclean to start, and the build will not write into it at all. +Hence, you can continue editing and compiling your Sage development +tree even while Docker builds are running. + + +Debugging a portability bug using Docker +---------------------------------------- + +Let us do another partial build. We choose a package that we suspect +might not work on all platforms, ``surf``, which was marked as +"experimental" in 2017:: + + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg BASE_IMAGE=ubuntu:latest \ + --build-arg NUMPROC=4 \ + --build-arg TARGETS_PRE=surf \ + --build-arg TARGETS=surf + Sending build context to Docker daemon 285MB + Step 1/28 : ARG BASE_IMAGE=ubuntu:latest + Step 2/28 : FROM ${BASE_IMAGE} + ---> 549b9b86cb8d + ... + Step 24/28 : ARG TARGETS_PRE="sagelib-build-deps" + ---> Running in 17d0ddb5ad7b + Removing intermediate container 17d0ddb5ad7b + ---> 7b51411520c3 + Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE} + ---> Running in 61833bea6a6d + make -j4 build/make/Makefile --stop + ... + [surf-1.0.6-gcc6] Attempting to download package surf-1.0.6-gcc6.tar.gz from mirrors + ... + [surf-1.0.6-gcc6] http://mirrors.mit.edu/sage/spkg/upstream/surf/surf-1.0.6-gcc6.tar.gz + [surf-1.0.6-gcc6] [......................................................................] + [surf-1.0.6-gcc6] surf-1.0.6-gcc6 + [surf-1.0.6-gcc6] ==================================================== + [surf-1.0.6-gcc6] Setting up build directory for surf-1.0.6-gcc6 + ... + [surf-1.0.6-gcc6] /usr/bin/ld: cannot find -lfl + [surf-1.0.6-gcc6] collect2: error: ld returned 1 exit status + [surf-1.0.6-gcc6] Makefile:504: recipe for target 'surf' failed + [surf-1.0.6-gcc6] make[3]: *** [surf] Error 1 + ... + [surf-1.0.6-gcc6] ************************************************************************ + [surf-1.0.6-gcc6] Error installing package surf-1.0.6-gcc6 + [surf-1.0.6-gcc6] ************************************************************************ + ... + Makefile:2088: recipe for target '/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6' failed + make[1]: *** [/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6] Error 1 + make[1]: Target 'surf' not remade because of errors. + make[1]: Leaving directory '/sage/build/make' + ... + Error building Sage. + The following package(s) may have failed to build (not necessarily + during this run of 'make surf'): + + * package: surf-1.0.6-gcc6 + last build time: Mar 26 22:07 + log file: /sage/logs/pkgs/surf-1.0.6-gcc6.log + build directory: /sage/local/var/tmp/sage/build/surf-1.0.6-gcc6 + + ... + Makefile:31: recipe for target 'surf' failed + make: *** [surf] Error 1 + The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE}' returned a non-zero code: 2 + +Note that no image id is shown at the end; the build failed, and no +image is created. However, the container in which the last step of +the build was attempted exists:: + + [mkoeppe@sage sage]$ docker ps -a |head -n3 + CONTAINER ID IMAGE COMMAND CREATED STATUS + 61833bea6a6d 7b51411520c3 "/bin/sh -c 'make SA…" 9 minutes ago Exited (2) 1 minute ago + 73987568712c ubuntu-latest-minimal-17 "/bin/bash" 24 hours ago Exited (0) 23 hours ago + +We can copy the build directory from the container for inspection:: + + [mkoeppe@sage sage]$ docker cp 61833bea6a6d:/sage/local/var/tmp/sage/build ubuntu-build + [mkoeppe@sage sage]$ ls ubuntu-build/surf*/src + AUTHORS TODO curve misc + COPYING acinclude.m4 debug missing + ChangeLog aclocal.m4 dither mkinstalldirs + INSTALL background.pic docs mt + Makefile config.guess draw src + Makefile.am config.log drawfunc surf.1 + Makefile.global config.status examples surf.xpm + Makefile.in config.sub gtkgui yaccsrc + NEWS configure image-formats + README configure.in install-sh + +Alternatively, we can use ``docker commit`` as explained earlier to +create an image from the container:: + + [mkoeppe@sage sage]$ docker commit 61833bea6a6d + sha256:003fbd511016fe305bd8494bb1747f0fbf4cb2c788b4e755e9099d9f2014a60d + [mkoeppe@sage sage]$ docker run -it 003fbd511 bash + root@2d9ac65f4572:/sage# (cd /sage/local/var/tmp/sage/build/surf* && /sage/sage --buildsh) + + Starting subshell with Sage environment variables set. Don't forget + to exit when you are done. Beware: + * Do not do anything with other copies of Sage on your system. + * Do not use this for installing Sage packages using "sage -i" or for + running "make" at Sage's root directory. These should be done + outside the Sage shell. + + Bypassing shell configuration files... + + Note: SAGE_ROOT=/sage + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ls /usr/lib/libfl* + /usr/lib/libflint-2.5.2.so /usr/lib/libflint-2.5.2.so.13.5.2 /usr/lib/libflint.a /usr/lib/libflint.so + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get update && apt-get install apt-file + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file update + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file search "/usr/lib/libfl.a" + flex-old: /usr/lib/libfl.a + freebsd-buildutils: /usr/lib/libfl.a + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get install flex-old + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ./spkg-install + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + /usr/bin/install -c surf /sage/local/bin/surf + /usr/bin/install -c -m 644 ./surf.1 /sage/local/share/man/man1/surf.1 + make[3]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' + make[2]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' + make[1]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ exit + root@2d9ac65f4572:/sage# exit + [mkoeppe@sage sage]$ + +A standard case of bitrot. + Symbolic names of system configurations --------------------------------------- From 2ba6637b9872d1f9c8038fadef4b2c88bb2395b4 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 20:05:29 -0400 Subject: [PATCH 08/22] reformatting --- src/doc/en/developer/portability_testing.rst | 29 +++++++++++--------- 1 file changed, 16 insertions(+), 13 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 1135ff43cdb..28385999b30 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -21,8 +21,9 @@ spectrum of platforms. Sage patchbots ============== -The `Sage patchbots `_ will automatically test your Trac ticket -by attempting an incremental build of Sage and running doctests. +The `Sage patchbots `_ will +automatically test your Trac ticket by attempting an incremental build +of Sage and running doctests. Sage buildbots ============== @@ -35,10 +36,10 @@ Developers' and users' tests on sage-release ============================================ Sage developers and users are encouraged to contribute to testing -releases that are announced on -`Sage Release `_ on their machines -and to report test results (success and failures) by responding to the -announcements. +releases that are announced on `Sage Release +`_ on their +machines and to report test results (success and failures) by +responding to the announcements. Testing Sage on a different platform using Docker ================================================= @@ -49,9 +50,9 @@ kernel. Using Docker Desktop for Mac and Windows, the containers can also be run on these platforms. All major Linux distributions provide ready-to-use Docker images, -which are published via `Docker Hub `_. For example, to run -the current stable (LTS) version of Ubuntu interactively, you can use -the shell command:: +which are published via `Docker Hub `_. For +example, to run the current stable (LTS) version of Ubuntu +interactively, you can use the shell command:: [mkoeppe@sage sage]$ docker run -it ubuntu:latest root@9f3398da43c2:/# @@ -93,8 +94,8 @@ This is not bootstrapped (``configure`` is missing), so let's bootstrap it:: [mkoeppe@sage worktree-ubuntu-latest] make configure ... -We can start a container again with same image, ``ubuntu:latest``, but this time let's -mount the current directory into it:: +We can start a container again with same image, ``ubuntu:latest``, but +this time let's mount the current directory into it:: [mkoeppe@sage worktree-ubuntu-latest]$ docker run -it --mount type=bind,source=$(pwd),target=/sage ubuntu:latest root@39d693b2a75d:/# mount | grep sage @@ -560,7 +561,9 @@ Symbolic names of system configurations To facilitate communication about host system configurations and for automating testing, Sage defines symbolic names composed of several -`Tox "factors" `_ in the file ``$SAGE_ROOT/tox.ini``. +`Tox "factors" +`_ +in the file ``$SAGE_ROOT/tox.ini``. The **system factor** describes a base operating system image. @@ -594,7 +597,7 @@ such as ``debian-buster-standard`` and ``centos-7-i386-minimal``. Automatic Docker-based build testing using tox ---------------------------------------------- -``tox`` is Python package that is widely used for automating tests of +``tox`` is a Python package that is widely used for automating tests of Python projects. Install ``tox`` for use with your system Python, for example using:: From 5ec6aedbed18f62c7d1530ef8bbb03883e3275f5 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 21:21:52 -0400 Subject: [PATCH 09/22] Add documentation on tox -e local --- src/doc/en/developer/portability_testing.rst | 145 ++++++++++++++++++- 1 file changed, 140 insertions(+), 5 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 28385999b30..93a0d9073bc 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -610,11 +610,9 @@ that we introduced above, there are the following factors: The **technology** factor describes how the environment is run: -- ``docker`` builds a Docker image as described above +- ``docker`` builds a Docker image as described above. -.. - - ``local`` runs testing on the host OS instead. This is currently only - implemented for ``local-homebrew-macos``; see below. +- ``local`` runs testing on the host OS instead. See below. The **configuration** factor (allowed to be empty): @@ -634,6 +632,143 @@ a silent build (``make V=0``), use:: [mkoeppe@sage sage]$ EXTRA_DOCKER_BUILD_ARGS="--build-arg USE_MAKEFLAGS=\"V=0\"" \ tox -e docker-ubuntu-bionic-standard +Automatic build testing on the host OS using tox -e local-direct +---------------------------------------------------------------- + +The ``local`` technology runs testing on the host OS instead. + +In contrast to the ``docker`` technology, it does not make a copy of +the source tree. It is most straightforward to run it from a +separate, distclean git worktree. + +Let us try a first variant of the ``local`` technology, the tox +environment called ``local-direct``. Because all builds with tox +begin by bootstrapping the source tree, you will need autotools and +other prerequisites installed in your system. See +``build/pkgs/*-bootstrap.txt`` for a list of system packages that +provide these prerequisites. + +We start by creating a fresh (distclean) git worktree. + + [mkoeppe@sage sage] git worktree add worktree-local + [mkoeppe@sage sage] cd worktree-local + [mkoeppe@sage worktree-local] ls + COPYING.txt ... Makefile ... configure.ac ... src tox.ini + +Again we build only a small package. Build targets can be passed as +positional arguments (separated from tox options by ``--``):: + + [mkoeppe@sage worktree-local] tox -e local-direct -- ratpoints + local-direct create: /Users/mkoeppe/.../worktree-local/.tox/local-direct + local-direct run-test-pre: PYTHONHASHSEED='2211987514' + ... + rm -rf config configure build/make/Makefile-auto.in + rm -f src/doc/en/installation/*.txt + src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt and src/doc/en/installation/debian-optional.txt + src/doc/bootstrap:48: installing src/doc/en/installation/fedora.txt and src/doc/en/installation/fedora-optional.txt + src/doc/bootstrap:48: installing src/doc/en/installation/cygwin.txt and src/doc/en/installation/cygwin-optional.txt + bootstrap:69: installing 'config/config.rpath' + configure.ac:328: installing 'config/compile' + configure.ac:113: installing 'config/config.guess' + configure.ac:113: installing 'config/config.sub' + configure.ac:68: installing 'config/install-sh' + configure.ac:68: installing 'config/missing' + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '/Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log' + [ratpoints-2.1.3.p5] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log + [ratpoints-2.1.3.p5] successfully installed. + [ratpoints-2.1.3.p5] build times: + ... + local-direct: commands succeeded + congratulations :) + +Let's investigate what happened here:: + + [mkoeppe@sage worktree-local]$ ls -l + total 2576 + -rw-r--r-- 1 mkoeppe staff 66501 Mar 26 20:34 COPYING.txt + -rw-r--r-- 1 mkoeppe staff 7542 Mar 26 20:41 Makefile + -rw-r--r-- 1 mkoeppe staff 19862 Mar 26 20:34 README.md + -rw-r--r-- 1 mkoeppe staff 53 Mar 26 20:34 VERSION.txt + -rw-r--r-- 1 mkoeppe staff 133587 Mar 26 20:45 aclocal.m4 + drwxr-xr-x 7 mkoeppe staff 224 Mar 26 20:45 autom4te.cache + -rwxr-xr-x 1 mkoeppe staff 7583 Mar 26 20:34 bootstrap + drwxr-xr-x 11 mkoeppe staff 352 Mar 26 20:34 build + -rw-r--r-- 1 mkoeppe staff 223 Mar 26 20:34 condarc.yml + drwxr-xr-x 8 mkoeppe staff 256 Mar 26 20:45 config + lrwxr-xr-x 1 mkoeppe staff 114 Mar 26 20:45 config.log -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/log/config.log + -rwxr-xr-x 1 mkoeppe staff 90411 Mar 26 20:46 config.status + -rwxr-xr-x 1 mkoeppe staff 887180 Mar 26 20:45 configure + -rw-r--r-- 1 mkoeppe staff 17070 Mar 26 20:41 configure.ac + drwxr-xr-x 8 mkoeppe staff 256 Mar 26 20:34 docker + lrwxr-xr-x 1 mkoeppe staff 103 Mar 26 20:45 logs -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/log + drwxr-xr-x 24 mkoeppe staff 768 Mar 26 20:45 m4 + lrwxr-xr-x 1 mkoeppe staff 105 Mar 26 20:45 prefix -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/local + -rwxr-xr-x 1 mkoeppe staff 4868 Mar 26 20:34 sage + drwxr-xr-x 16 mkoeppe staff 512 Mar 26 20:46 src + -rw-r--r-- 1 mkoeppe staff 13478 Mar 26 20:41 tox.ini + drwxr-xr-x 4 mkoeppe staff 128 Mar 26 20:46 upstream + +There is no ``local`` subdirectory. This is part of a strategy to +keep the source tree clean to the extent possible. In particular: + +- ``tox`` configured the build to use a separate ``$SAGE_LOCAL`` + hierarchy in a directory under the tox environment directory + ``.tox/local-direct``. It created a symbolic link ``prefix`` that + points there, for convenience:: + + [mkoeppe@sage worktree-local]$ ls -l prefix/lib/*rat* + -rw-r--r-- 1 mkoeppe staff 165968 Mar 26 20:46 prefix/lib/libratpoints.a + +- Likewise, it created a separate ``logs`` directory, again under the + tox environment directory, and a symbolic link. + +This makes it possible for advanced users to test several ``local`` +tox environments (such as ``local-direct`` and +``local-direct-python2``) out of one worktree. However, because a +build still writes configuration scripts and build artefacts (such as +``config.status``) into the worktree, only one ``local`` build can run +at a time in a given worktree. + +The tox environment directory will be reused for the next ``tox`` run, +which will therefore do an incremental build. To start a fresh build, +you can use the ``-r`` option. + +Automatic build testing on the host OS with best-effort isolation using tox -e local +------------------------------------------------------------------------------------ + +``tox -e local`` (without ``-direct``) attempts a best-effort +isolation from the user's environment as follows: + +- All environment variables are set to standard values; with the + exception of ``MAKE`` and ``EXTRA_CONFIGURE_ARGS``. In particular, + ``PATH`` is set to just ``/usr/bin:/bin:/usr/sbin:/sbin``; it does + not include ``/usr/local/bin``. + + +Note, however, that various packages have build scripts that use +``/usr/local`` or other popular file system locations such as +``/opt/sfw/``. Therefore, the isolation is not complete. Using +``/usr/local`` is considered standard behavior. On the other hand, we +consider a package build script that inspects other file system +locations to be a bug of the Sage distribution, which should be +reported and fixed on a ticket. + + +.. + Automatic build testing on macOS with best-effort isolated installation of Homebrew + ----------------------------------------------------------------------------------- + + adds toxenvs local-homebrew-macos-{minimal,standard} for testing with (best-effort) isolated installations of homebrew on macOS (not using docker). + + To test: + + + This is currently only + implemented for ``local-homebrew-macos``; see below. + Automatic parallel tox runs on GitHub Actions --------------------------------------------- @@ -652,7 +787,7 @@ GitHub Actions runs these build jobs on 2-core machines with 7 GB of RAM memory and 14 GB of SSD disk space, cf. `here `_, and has a time limit of 6h per job. This is just barely enough for a -typical ``minimal`` build followed by make ptest to succeed; and +typical ``minimal`` build followed by ``make ptest`` to succeed; and plenty of time for a typical ``standard`` build to succeed. From 91a54bf38b45f693a3f6e1cd97b88d0bda61c1ad Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 22:37:11 -0400 Subject: [PATCH 10/22] Add documentation of local-homebrew-macos --- src/doc/en/developer/portability_testing.rst | 78 +++++++++++++++++--- 1 file changed, 69 insertions(+), 9 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 93a0d9073bc..ba6768220bf 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -679,7 +679,6 @@ positional arguments (separated from tox options by ``--``):: sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '/Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log' [ratpoints-2.1.3.p5] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log [ratpoints-2.1.3.p5] successfully installed. - [ratpoints-2.1.3.p5] build times: ... local-direct: commands succeeded congratulations :) @@ -757,18 +756,79 @@ locations to be a bug of the Sage distribution, which should be reported and fixed on a ticket. -.. - Automatic build testing on macOS with best-effort isolated installation of Homebrew - ----------------------------------------------------------------------------------- +Automatic build testing on macOS with a best-effort isolated installation of Homebrew +------------------------------------------------------------------------------------- - adds toxenvs local-homebrew-macos-{minimal,standard} for testing with (best-effort) isolated installations of homebrew on macOS (not using docker). +XCode on macOS does not provide the prerequisites for bootstrapping +the Sage distribution. A good way to install them is using the +Homebrew package manager. - To test: +In fact, Sage provides a tox environment that automatically installs +an isolated copy of Homebrew with all prerequisites for bootstrapping:: + [mkoeppe@sage worktree-local]$ tox -e local-homebrew-macos-minimal -- lrslib + local-homebrew-macos-minimal create: /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal + local-homebrew-macos-minimal run-test-pre: PYTHONHASHSEED='4246149402' + ... + Initialized empty Git repository in /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/.git/ + ... + Tapped 2 commands and 4942 formulae (5,205 files, 310.7MB). + ==> Downloading https://ftp.gnu.org/gnu/gettext/gettext-0.20.1.tar.xz + ... + ==> Pouring autoconf-2.69.catalina.bottle.4.tar.gz + ... + ==> Pouring pkg-config-0.29.2.catalina.bottle.1.tar.gz + /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/Cellar/pkg-config/0.29.2: 11 files, 623.4KB + ==> Caveats + ==> gettext + gettext is keg-only, which means it was not symlinked into /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew, + because macOS provides the BSD gettext library & some software gets confused if both are in the library path. + + If you need to have gettext first in your PATH run: + echo 'export PATH="/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/bin:$PATH"' >> ~/.bash_profile + + For compilers to find gettext you may need to set: + export LDFLAGS="-L/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib" + export CPPFLAGS="-I/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include" + ... + local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 base-toolchain && make -k V=0 SAGE_SPKG="sage-spkg -y -o" SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!python2,!nose,!pathpy,!gap,!cysignals,!linbox,!git,!ppl" lrslib' + ... + bootstrap:69: installing 'config/config.rpath' + ... + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll gf2x gfan gfortran givaro glpk gp2c gsl lcalc libatomic_ops libsemigroups lrcalc m4ri m4rie mpfi nauty ninja_build ntl openblas pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small patch perl_term_readline_gnu planarity r readline rw symmetrica tachyon xz yasm zeromq + checking for the package system in use... homebrew + configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: + configure: $ brew install cmake gcc gsl mpfi ninja openblas gpatch r readline xz yasm zeromq + configure: After installation, re-run configure using: + configure: $ ./config.status --recheck && ./config.status + ... + sage-logger -p 'sage-spkg -y -o lrslib-062+autotools-2017-03-03.p1' '/Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log' + [lrslib-062+autotools-2017-03-03.p1] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log + [lrslib-062+autotools-2017-03-03.p1] successfully installed. + ... + local-homebrew-macos-minimal: commands succeeded + congratulations :) + +The tox environment uses the subdirectory ``homebrew`` of the +environment directory ``.tox/local-homebrew-macos-minimal`` as the +Homebrew prefix. This installation does not interact in any way with +a Homebrew installation in ``/usr/local`` that you may have. + +The test script sets the ``PATH`` to the ``bin`` directory of the +Homebrew prefix, followed by ``/usr/bin:/bin:/usr/sbin:/sbin``. It +then uses the script ``$SAGE_ROOT/.homebrew-build-env`` to set +environment variables so that Sage's build scripts will find +"keg-only" packages such as ``gettext``. + +The ``local-homebrew-macos-standard`` environment additionally +installs (in its separate isolated copy of Homebrew) all Homebrew +packages known to Sage for which an ``spkg-configure.m4`` is +available; this is similar to the ``docker`` tox environments +described earlier. - This is currently only - implemented for ``local-homebrew-macos``; see below. - Automatic parallel tox runs on GitHub Actions --------------------------------------------- From 7eac94a28506cab19de282dd709593e686b91b43 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 22:58:20 -0400 Subject: [PATCH 11/22] more on GitHub Actions --- src/doc/en/developer/portability_testing.rst | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index ba6768220bf..0977d53c678 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -850,6 +850,20 @@ and has a time limit of 6h per job. This is just barely enough for a typical ``minimal`` build followed by ``make ptest`` to succeed; and plenty of time for a typical ``standard`` build to succeed. +Build logs become available as "artifacts" when all jobs of the +workflow have finished. Each job generates one tarball. +"Annotations" highlight certain top-level errors or warnings issued +during the build. + +The following procedure seems to work well for testing branches during +development. Create a branch from a recent beta release that contains +the default GitHub Actions configuration; name it ``TESTER``, say. +Edit ``$SAGE_ROOT/.github/workflows/tox.yml`` to include the system +configurations that you wish to test. Commit and push the branch to +your GitHub fork of sage. Next, push your development branch to your +GitHub repository and create a pull request against the ``TESTER`` +branch. This will trigger the GitHub Actions workflow. + .. Workflow for responding to portability bug reports From 426aec1e2c34052f871429460ca929d87e10de32 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Thu, 26 Mar 2020 23:13:16 -0400 Subject: [PATCH 12/22] More explanation regarding mac / linux --- src/doc/en/developer/portability_testing.rst | 19 +++++++++++++++---- 1 file changed, 15 insertions(+), 4 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 0977d53c678..1b36643c85c 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -44,10 +44,21 @@ responding to the announcements. Testing Sage on a different platform using Docker ================================================= -`Docker `_ is a popular virtualization software, -running Linux operating system images in containers on a shared Linux -kernel. Using Docker Desktop for Mac and Windows, the containers can -also be run on these platforms. +`Docker `_ is a popular virtualization +software, running Linux operating system images ("Docker images") in +containers on a shared Linux kernel. These containers can be run +using a Docker client on your Linux, Mac, or Windows box, as well as +on various cloud services. + +To get started, you need to install a `Docker client +`. The clients are available for +Linux, Mac, and Windows. The clients for the latter are known as +"Docker Desktop". + +All examples in this section were obtained using Docker Desktop for +Mac; but the `command-line user interface +` for the +other platforms is identical. All major Linux distributions provide ready-to-use Docker images, which are published via `Docker Hub `_. For From 43195f63d1a35d420a8bc1c8ebf60856adfcf468 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 09:49:19 -0400 Subject: [PATCH 13/22] add links to docker docs --- src/doc/en/developer/portability_testing.rst | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 1b36643c85c..487c4e10578 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -288,7 +288,10 @@ have no access to the worktree:: The image ``ubuntu-latest-minimal-17`` can be run in as many containers as we want and can also be shared with other users or developers so that they can run it in a container on their machine. -(See the Docker documentation on how to do this.) +(See the Docker documentation on how to `share images on Docker Hub +` or to `save images to a +tar archive +`.) This facilitates collaboration on fixing portability bugs of the Sage distribution. After reproducing a portability bug on a container, From 3d2c3faf33a85e0402c73d98fd10917ab25c70dd Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 09:55:01 -0400 Subject: [PATCH 14/22] add crossref to Installation Guide --- src/doc/en/developer/portability_testing.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 487c4e10578..ccc4cb56f92 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -195,6 +195,10 @@ on our container to install the necessary build prerequisites:: ... Done. +(The Sage `Installation Guide <../installation/index.html>`_ also +provides such command lines for some distributions; these are +automatically generated from the database of package names.) + Now we can start the build:: root@39d693b2a75d:/sage# ./configure From 61539457b4094c1f28935d23dff9ddc6736c5d9b Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 10:07:59 -0400 Subject: [PATCH 15/22] merge the sections on tox factors --- src/doc/en/developer/portability_testing.rst | 63 +++++++++++--------- 1 file changed, 34 insertions(+), 29 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index ccc4cb56f92..b8545d4aac6 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -579,12 +579,33 @@ Symbolic names of system configurations To facilitate communication about host system configurations and for automating testing, Sage defines symbolic names composed of several -`Tox "factors" -`_ -in the file ``$SAGE_ROOT/tox.ini``. -The **system factor** describes a base operating system -image. + + +Automatic Docker-based build testing using tox +---------------------------------------------- + +`tox ` is a Python package that +is widely used for automating tests of Python projects. + +Install ``tox`` for use with your system Python, for example using:: + + [mkoeppe@sage sage]$ pip install --user tox + +A tox "environment" is identified by a symbolic name composed of +several `Tox "factors" +`_, +which are defined in the file ``$SAGE_ROOT/tox.ini``. + +The **technology** factor describes how the environment is run: + +- ``docker`` builds a Docker image as described above. + +- ``local`` runs testing on the host OS instead. We explain this + technology in a later section. + +The next two factors determine the host system configuration: The +**system factor** describes a base operating system image. - Examples are ``ubuntu-focal``, ``debian-buster``, ``archlinux-latest``, ``fedora-30``, ``slackware-14.2``, @@ -611,34 +632,18 @@ installed on the system before building Sage: The factors are connected by a hyphen to name a system configuration, such as ``debian-buster-standard`` and ``centos-7-i386-minimal``. - -Automatic Docker-based build testing using tox ----------------------------------------------- - -``tox`` is a Python package that is widely used for automating tests of -Python projects. - -Install ``tox`` for use with your system Python, for example using:: - - [mkoeppe@sage sage]$ pip install --user tox - -A tox "environment" is identified by a name composed of several -factors. In addition to the system factor and the packages factor -that we introduced above, there are the following factors: - -The **technology** factor describes how the environment is run: - -- ``docker`` builds a Docker image as described above. - -- ``local`` runs testing on the host OS instead. See below. - -The **configuration** factor (allowed to be empty): +Finally, the **configuration** factor (which is allowed to be empty) +controls how the ``configure`` script is run. - ``python2`` adds the argument ``--with-python=2`` to the ``configure`` run. -The factors are connected by a hyphen to name a tox environment. To -run an environment:: +The factors are connected by a hyphen to name a tox environment. (The +order of the factors does not matter; however, for consistency and +because the ordered name is used for caching purposes, we recommend to +use the factors in the listed order.) + +To run an environment:: [mkoeppe@sage sage]$ tox -e docker-slackware-14.2-minimal [mkoeppe@sage sage]$ tox -e docker-ubuntu-bionic-standard-python2 From ab11ebd0c2c85846aea64064cebd3229cec56e3b Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 10:11:52 -0400 Subject: [PATCH 16/22] elide some details from terminal output --- src/doc/en/developer/portability_testing.rst | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index b8545d4aac6..b203cabae3c 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -814,19 +814,17 @@ an isolated copy of Homebrew with all prerequisites for bootstrapping:: export LDFLAGS="-L/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib" export CPPFLAGS="-I/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include" ... - local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 base-toolchain && make -k V=0 SAGE_SPKG="sage-spkg -y -o" SAGE_CHECK=warn SAGE_CHECK_PACKAGES="!cython,!r,!python3,!python2,!nose,!pathpy,!gap,!cysignals,!linbox,!git,!ppl" lrslib' + local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 ... lrslib' ... bootstrap:69: installing 'config/config.rpath' ... checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes ... - configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer cmake eclib ecm flint fplll gf2x gfan gfortran givaro glpk gp2c gsl lcalc libatomic_ops libsemigroups lrcalc m4ri m4rie mpfi nauty ninja_build ntl openblas pari pari_elldata pari_galdata pari_galpol pari_nftables pari_seadata pari_seadata_small patch perl_term_readline_gnu planarity r readline rw symmetrica tachyon xz yasm zeromq + configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer ... tachyon xz yasm zeromq checking for the package system in use... homebrew configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: configure: $ brew install cmake gcc gsl mpfi ninja openblas gpatch r readline xz yasm zeromq - configure: After installation, re-run configure using: - configure: $ ./config.status --recheck && ./config.status ... sage-logger -p 'sage-spkg -y -o lrslib-062+autotools-2017-03-03.p1' '/Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log' [lrslib-062+autotools-2017-03-03.p1] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log @@ -848,9 +846,9 @@ environment variables so that Sage's build scripts will find The ``local-homebrew-macos-standard`` environment additionally installs (in its separate isolated copy of Homebrew) all Homebrew -packages known to Sage for which an ``spkg-configure.m4`` is -available; this is similar to the ``docker`` tox environments -described earlier. +packages known to Sage for which the ``spkg-configure.m4`` mechanism +is implemented; this is similar to the ``docker-standard`` tox +environments described earlier. Automatic parallel tox runs on GitHub Actions From c7fcf9184934c05d59f556bc110554111459fac7 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 10:18:17 -0400 Subject: [PATCH 17/22] change 'ls -l' to 'ls -la' and elide some details --- src/doc/en/developer/portability_testing.rst | 30 +++++++++++--------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index b203cabae3c..065d8cc966d 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -708,26 +708,28 @@ positional arguments (separated from tox options by ``--``):: Let's investigate what happened here:: - [mkoeppe@sage worktree-local]$ ls -l + [mkoeppe@sage worktree-local]$ ls -la total 2576 - -rw-r--r-- 1 mkoeppe staff 66501 Mar 26 20:34 COPYING.txt + drwxr-xr-x 35 mkoeppe staff 1120 Mar 26 22:20 . + drwxr-xr-x 63 mkoeppe staff 2016 Mar 27 09:35 .. + ... + lrwxr-xr-x 1 mkoeppe staff 10 Mar 26 20:34 .dockerignore -> .gitignore + -rw-r--r-- 1 mkoeppe staff 74 Mar 26 20:34 .git + ... + -rw-r--r-- 1 mkoeppe staff 1212 Mar 26 20:41 .gitignore + ... + drwxr-xr-x 7 mkoeppe staff 224 Mar 26 22:11 .tox + ... -rw-r--r-- 1 mkoeppe staff 7542 Mar 26 20:41 Makefile - -rw-r--r-- 1 mkoeppe staff 19862 Mar 26 20:34 README.md - -rw-r--r-- 1 mkoeppe staff 53 Mar 26 20:34 VERSION.txt - -rw-r--r-- 1 mkoeppe staff 133587 Mar 26 20:45 aclocal.m4 - drwxr-xr-x 7 mkoeppe staff 224 Mar 26 20:45 autom4te.cache - -rwxr-xr-x 1 mkoeppe staff 7583 Mar 26 20:34 bootstrap - drwxr-xr-x 11 mkoeppe staff 352 Mar 26 20:34 build - -rw-r--r-- 1 mkoeppe staff 223 Mar 26 20:34 condarc.yml - drwxr-xr-x 8 mkoeppe staff 256 Mar 26 20:45 config - lrwxr-xr-x 1 mkoeppe staff 114 Mar 26 20:45 config.log -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/log/config.log + ... + lrwxr-xr-x 1 mkoeppe staff 114 Mar 26 20:45 config.log -> .tox/local-direct/log/config.log -rwxr-xr-x 1 mkoeppe staff 90411 Mar 26 20:46 config.status -rwxr-xr-x 1 mkoeppe staff 887180 Mar 26 20:45 configure -rw-r--r-- 1 mkoeppe staff 17070 Mar 26 20:41 configure.ac - drwxr-xr-x 8 mkoeppe staff 256 Mar 26 20:34 docker - lrwxr-xr-x 1 mkoeppe staff 103 Mar 26 20:45 logs -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/log + ... + lrwxr-xr-x 1 mkoeppe staff 103 Mar 26 20:45 logs -> .tox/local-direct/log drwxr-xr-x 24 mkoeppe staff 768 Mar 26 20:45 m4 - lrwxr-xr-x 1 mkoeppe staff 105 Mar 26 20:45 prefix -> /Users/mkoeppe/.../worktree-local/.tox/local-direct/local + lrwxr-xr-x 1 mkoeppe staff 105 Mar 26 20:45 prefix -> .tox/local-direct/local -rwxr-xr-x 1 mkoeppe staff 4868 Mar 26 20:34 sage drwxr-xr-x 16 mkoeppe staff 512 Mar 26 20:46 src -rw-r--r-- 1 mkoeppe staff 13478 Mar 26 20:41 tox.ini From e73bc8df254d0355c4039b139e0f2a14b58802a3 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Fri, 27 Mar 2020 10:25:43 -0400 Subject: [PATCH 18/22] Elide more details, shorten directory names more --- src/doc/en/developer/portability_testing.rst | 55 +++++++------------- 1 file changed, 19 insertions(+), 36 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 065d8cc966d..214ea5d0682 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -351,7 +351,7 @@ Then, to bootstrap and configure...:: ADD src/bin src/bin ADD src/Makefile.in src/Makefile.in ARG EXTRA_CONFIGURE_ARGS="" - RUN ./configure --enable-build-as-root --enable-option-checking ${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1) + RUN ./configure --enable-build-as-root ${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1) Finally, to build and test...:: @@ -474,9 +474,7 @@ might not work on all platforms, ``surf``, which was marked as [surf-1.0.6-gcc6] Attempting to download package surf-1.0.6-gcc6.tar.gz from mirrors ... [surf-1.0.6-gcc6] http://mirrors.mit.edu/sage/spkg/upstream/surf/surf-1.0.6-gcc6.tar.gz - [surf-1.0.6-gcc6] [......................................................................] - [surf-1.0.6-gcc6] surf-1.0.6-gcc6 - [surf-1.0.6-gcc6] ==================================================== + ... [surf-1.0.6-gcc6] Setting up build directory for surf-1.0.6-gcc6 ... [surf-1.0.6-gcc6] /usr/bin/ld: cannot find -lfl @@ -484,9 +482,7 @@ might not work on all platforms, ``surf``, which was marked as [surf-1.0.6-gcc6] Makefile:504: recipe for target 'surf' failed [surf-1.0.6-gcc6] make[3]: *** [surf] Error 1 ... - [surf-1.0.6-gcc6] ************************************************************************ [surf-1.0.6-gcc6] Error installing package surf-1.0.6-gcc6 - [surf-1.0.6-gcc6] ************************************************************************ ... Makefile:2088: recipe for target '/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6' failed make[1]: *** [/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6] Error 1 @@ -541,14 +537,8 @@ create an image from the container:: root@2d9ac65f4572:/sage# (cd /sage/local/var/tmp/sage/build/surf* && /sage/sage --buildsh) Starting subshell with Sage environment variables set. Don't forget - to exit when you are done. Beware: - * Do not do anything with other copies of Sage on your system. - * Do not use this for installing Sage packages using "sage -i" or for - running "make" at Sage's root directory. These should be done - outside the Sage shell. - - Bypassing shell configuration files... - + to exit when you are done. + ... Note: SAGE_ROOT=/sage (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ls /usr/lib/libfl* /usr/lib/libflint-2.5.2.so /usr/lib/libflint-2.5.2.so.13.5.2 /usr/lib/libflint.a /usr/lib/libflint.so @@ -564,8 +554,7 @@ create an image from the container:: ... /usr/bin/install -c surf /sage/local/bin/surf /usr/bin/install -c -m 644 ./surf.1 /sage/local/share/man/man1/surf.1 - make[3]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' - make[2]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' + ... make[1]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ exit root@2d9ac65f4572:/sage# exit @@ -685,22 +674,16 @@ positional arguments (separated from tox options by ``--``):: local-direct create: /Users/mkoeppe/.../worktree-local/.tox/local-direct local-direct run-test-pre: PYTHONHASHSEED='2211987514' ... - rm -rf config configure build/make/Makefile-auto.in - rm -f src/doc/en/installation/*.txt - src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt and src/doc/en/installation/debian-optional.txt - src/doc/bootstrap:48: installing src/doc/en/installation/fedora.txt and src/doc/en/installation/fedora-optional.txt - src/doc/bootstrap:48: installing src/doc/en/installation/cygwin.txt and src/doc/en/installation/cygwin-optional.txt + src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt... bootstrap:69: installing 'config/config.rpath' configure.ac:328: installing 'config/compile' configure.ac:113: installing 'config/config.guess' - configure.ac:113: installing 'config/config.sub' - configure.ac:68: installing 'config/install-sh' - configure.ac:68: installing 'config/missing' + ... checking for a BSD-compatible install... /usr/bin/install -c checking whether build environment is sane... yes ... - sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '/Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log' - [ratpoints-2.1.3.p5] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log + sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log' + [ratpoints-2.1.3.p5] installing. Log file: .../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log [ratpoints-2.1.3.p5] successfully installed. ... local-direct: commands succeeded @@ -792,10 +775,10 @@ In fact, Sage provides a tox environment that automatically installs an isolated copy of Homebrew with all prerequisites for bootstrapping:: [mkoeppe@sage worktree-local]$ tox -e local-homebrew-macos-minimal -- lrslib - local-homebrew-macos-minimal create: /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal + local-homebrew-macos-minimal create: .../worktree-local/.tox/local-homebrew-macos-minimal local-homebrew-macos-minimal run-test-pre: PYTHONHASHSEED='4246149402' ... - Initialized empty Git repository in /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/.git/ + Initialized empty Git repository in .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/.git/ ... Tapped 2 commands and 4942 formulae (5,205 files, 310.7MB). ==> Downloading https://ftp.gnu.org/gnu/gettext/gettext-0.20.1.tar.xz @@ -803,20 +786,20 @@ an isolated copy of Homebrew with all prerequisites for bootstrapping:: ==> Pouring autoconf-2.69.catalina.bottle.4.tar.gz ... ==> Pouring pkg-config-0.29.2.catalina.bottle.1.tar.gz - /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/Cellar/pkg-config/0.29.2: 11 files, 623.4KB + .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/Cellar/pkg-config/0.29.2: 11 files, 623.4KB ==> Caveats ==> gettext - gettext is keg-only, which means it was not symlinked into /Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew, + gettext is keg-only, which means it was not symlinked into .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew, because macOS provides the BSD gettext library & some software gets confused if both are in the library path. If you need to have gettext first in your PATH run: - echo 'export PATH="/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/bin:$PATH"' >> ~/.bash_profile + echo 'export PATH=".../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/bin:$PATH"' >> ~/.bash_profile For compilers to find gettext you may need to set: - export LDFLAGS="-L/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib" - export CPPFLAGS="-I/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include" + export LDFLAGS="-L.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib" + export CPPFLAGS="-I.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include" ... - local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=/Users/mkoeppe/.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 ... lrslib' + local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 ... lrslib' ... bootstrap:69: installing 'config/config.rpath' ... @@ -828,8 +811,8 @@ an isolated copy of Homebrew with all prerequisites for bootstrapping:: configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: configure: $ brew install cmake gcc gsl mpfi ninja openblas gpatch r readline xz yasm zeromq ... - sage-logger -p 'sage-spkg -y -o lrslib-062+autotools-2017-03-03.p1' '/Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log' - [lrslib-062+autotools-2017-03-03.p1] installing. Log file: /Users/mkoeppe/.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log + sage-logger -p 'sage-spkg -y -o lrslib-062+autotools-2017-03-03.p1' '.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log' + [lrslib-062+autotools-2017-03-03.p1] installing. Log file: .../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log [lrslib-062+autotools-2017-03-03.p1] successfully installed. ... local-homebrew-macos-minimal: commands succeeded From a31b2a0effbe0442dffd65fc00ae1b8f116690eb Mon Sep 17 00:00:00 2001 From: Dima Pasechnik Date: Fri, 27 Mar 2020 23:46:25 -0400 Subject: [PATCH 19/22] Reviewer patch --- src/doc/en/developer/portability_testing.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 214ea5d0682..05fb991ef63 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -293,9 +293,9 @@ The image ``ubuntu-latest-minimal-17`` can be run in as many containers as we want and can also be shared with other users or developers so that they can run it in a container on their machine. (See the Docker documentation on how to `share images on Docker Hub -` or to `save images to a +`_ or to `save images to a tar archive -`.) +`_.) This facilitates collaboration on fixing portability bugs of the Sage distribution. After reproducing a portability bug on a container, From b7b45fe9a62a533db9f4199da5a57e0afd4d4139 Mon Sep 17 00:00:00 2001 From: Dima Pasechnik Date: Sat, 28 Mar 2020 01:00:45 -0400 Subject: [PATCH 20/22] Reviewer patch --- src/doc/en/developer/portability_testing.rst | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 05fb991ef63..0d64887f4f7 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -51,13 +51,13 @@ using a Docker client on your Linux, Mac, or Windows box, as well as on various cloud services. To get started, you need to install a `Docker client -`. The clients are available for +`_. The clients are available for Linux, Mac, and Windows. The clients for the latter are known as "Docker Desktop". All examples in this section were obtained using Docker Desktop for Mac; but the `command-line user interface -` for the +`_ for the other platforms is identical. All major Linux distributions provide ready-to-use Docker images, @@ -574,7 +574,7 @@ automating testing, Sage defines symbolic names composed of several Automatic Docker-based build testing using tox ---------------------------------------------- -`tox ` is a Python package that +`tox `_ is a Python package that is widely used for automating tests of Python projects. Install ``tox`` for use with your system Python, for example using:: From f7d2dc9fd3975cba376e246797d7cee2b47a3af1 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sat, 28 Mar 2020 23:36:20 -0400 Subject: [PATCH 21/22] Remove old section head --- src/doc/en/developer/portability_testing.rst | 8 -------- 1 file changed, 8 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 0d64887f4f7..5e4719a6289 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -563,14 +563,6 @@ create an image from the container:: A standard case of bitrot. -Symbolic names of system configurations ---------------------------------------- - -To facilitate communication about host system configurations and for -automating testing, Sage defines symbolic names composed of several - - - Automatic Docker-based build testing using tox ---------------------------------------------- From 99aca39dea0226372a3f12aed3a13301a3380ff4 Mon Sep 17 00:00:00 2001 From: Matthias Koeppe Date: Sun, 29 Mar 2020 00:26:59 -0400 Subject: [PATCH 22/22] Reformat list, remove unfinished commented out section --- src/doc/en/developer/portability_testing.rst | 21 +++++++++----------- 1 file changed, 9 insertions(+), 12 deletions(-) diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst index 5e4719a6289..d60f10e3b18 100644 --- a/src/doc/en/developer/portability_testing.rst +++ b/src/doc/en/developer/portability_testing.rst @@ -854,20 +854,17 @@ workflow have finished. Each job generates one tarball. during the build. The following procedure seems to work well for testing branches during -development. Create a branch from a recent beta release that contains -the default GitHub Actions configuration; name it ``TESTER``, say. -Edit ``$SAGE_ROOT/.github/workflows/tox.yml`` to include the system -configurations that you wish to test. Commit and push the branch to -your GitHub fork of sage. Next, push your development branch to your -GitHub repository and create a pull request against the ``TESTER`` -branch. This will trigger the GitHub Actions workflow. +development: +- Create a branch from a recent beta release that contains the default + GitHub Actions configuration; name it ``TESTER``, say. -.. - Workflow for responding to portability bug reports - ================================================== +- Edit ``$SAGE_ROOT/.github/workflows/tox.yml`` to include the system + config wish to test. - 1. Reproduce the platform and the bug. +- Commit and push the branch to your GitHub fork of sage. - 2. Fix the bug. +- Push your development branch to your GitHub repository and create a + pull request against the ``TESTER`` branch. This will trigger the + GitHub Actions workflow.