diff --git a/src/doc/en/developer/index.rst b/src/doc/en/developer/index.rst index f0e027a4aa6..3b4e44ff4c5 100644 --- a/src/doc/en/developer/index.rst +++ b/src/doc/en/developer/index.rst @@ -126,6 +126,14 @@ Running Sage's tests doctesting +Testing on multiple platforms +----------------------------- + +.. toctree:: + :maxdepth: 3 + + portability_testing + Contributing to Manuals and Tutorials ------------------------------------- diff --git a/src/doc/en/developer/portability_testing.rst b/src/doc/en/developer/portability_testing.rst new file mode 100644 index 00000000000..d60f10e3b18 --- /dev/null +++ b/src/doc/en/developer/portability_testing.rst @@ -0,0 +1,870 @@ +.. nodoctest + +.. highlight:: shell-session + +.. _chapter-portability_testing: + +============================= +Testing on multiple platforms +============================= + +Sage is intended to build and run on a variety of platforms, +including all major Linux distributions, as well as MacOS, and +Windows (with Cygwin). + +There is considerable variation among these platforms. +To ensure that Sage continues to build correctly on users' +machines, it is crucial to test changes to Sage, in particular +when external packages are added or upgraded, on a wide +spectrum of platforms. + +Sage patchbots +============== + +The `Sage patchbots `_ will +automatically test your Trac ticket by attempting an incremental build +of Sage and running doctests. + +Sage buildbots +============== + +The `Sage Release buildbot `_ +builds entire tarballs (e.g., all the development releases) on a +variety of machines. + +Developers' and users' tests on sage-release +============================================ + +Sage developers and users are encouraged to contribute to testing +releases that are announced on `Sage Release +`_ on their +machines and to report test results (success and failures) by +responding to the announcements. + +Testing Sage on a different platform using Docker +================================================= + +`Docker `_ is a popular virtualization +software, running Linux operating system images ("Docker images") in +containers on a shared Linux kernel. These containers can be run +using a Docker client on your Linux, Mac, or Windows box, as well as +on various cloud services. + +To get started, you need to install a `Docker client +`_. The clients are available for +Linux, Mac, and Windows. The clients for the latter are known as +"Docker Desktop". + +All examples in this section were obtained using Docker Desktop for +Mac; but the `command-line user interface +`_ for the +other platforms is identical. + +All major Linux distributions provide ready-to-use Docker images, +which are published via `Docker Hub `_. For +example, to run the current stable (LTS) version of Ubuntu +interactively, you can use the shell command:: + + [mkoeppe@sage sage]$ docker run -it ubuntu:latest + root@9f3398da43c2:/# + +Here ``ubuntu`` is referred to as the "image (name)" and ``latest`` as +the "tag". Other releases of Ubuntu are available under different +tags, such as ``xenial`` or ``devel``. + +The above command drops you in a root shell on the container:: + + root@9f3398da43c2:/# uname -a + Linux 9f3398da43c2 4.19.76-linuxkit #1 SMP Thu Oct 17 19:31:58 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux + root@9f3398da43c2:/# df -h + Filesystem Size Used Avail Use% Mounted on + overlay 181G 116G 56G 68% / + tmpfs 64M 0 64M 0% /dev + tmpfs 2.7G 0 2.7G 0% /sys/fs/cgroup + shm 64M 0 64M 0% /dev/shm + /dev/sda1 181G 116G 56G 68% /etc/hosts + tmpfs 2.7G 0 2.7G 0% /proc/acpi + tmpfs 2.7G 0 2.7G 0% /sys/firmware + +Exiting the shell terminates the container:: + + root@9f3398da43c2:/# ^D + [mkoeppe@sage sage]$ + +Let us work with a distclean Sage source tree. If you are using git, +a good way to get one (without losing a precious installation in +``SAGE_LOCAL``) is by creating a new worktree:: + + [mkoeppe@sage sage] git worktree add worktree-ubuntu-latest + [mkoeppe@sage sage] cd worktree-ubuntu-latest + [mkoeppe@sage worktree-ubuntu-latest] ls + COPYING.txt ... Makefile ... configure.ac ... src tox.ini + +This is not bootstrapped (``configure`` is missing), so let's bootstrap it:: + + [mkoeppe@sage worktree-ubuntu-latest] make configure + ... + +We can start a container again with same image, ``ubuntu:latest``, but +this time let's mount the current directory into it:: + + [mkoeppe@sage worktree-ubuntu-latest]$ docker run -it --mount type=bind,source=$(pwd),target=/sage ubuntu:latest + root@39d693b2a75d:/# mount | grep sage + osxfs on /sage type fuse.osxfs (rw,nosuid,nodev,relatime,user_id=0,group_id=0,allow_other,max_read=1048576) + root@39d693b2a75d:/# cd sage + root@39d693b2a75d:/sage# ls + COPYING.txt ... Makefile ... config configure configure.ac ... src tox.ini + +Typical Docker images provide minimal installations of packages only:: + + root@39d693b2a75d:/sage# command -v python + root@39d693b2a75d:/sage# command -v gcc + root@39d693b2a75d:/sage# + +As you can see above, the image ``ubuntu:latest`` has neither a Python nor +a GCC installed, which are among the build prerequisites of Sage. We +need to install them using the distribution's package manager first. + +Sage facilitates testing various distributions on Docker as follows. + +Discovering the system's package system +--------------------------------------- + +:: + + root@39d693b2a75d:/sage# build/bin/sage-guess-package-system + debian + +Let's install gcc, hoping that the Ubuntu package providing it is +simply named ``gcc``. If we forgot what the package manager on +Debian-derived distributions is called, we can ask Sage for a +reminder:: + + root@39d693b2a75d:/sage# build/bin/sage-print-system-package-command debian install gcc + sudo apt-get install gcc + +We are already root, so we can drop the ``sudo``, of course. +And we remember that we need to fetch the current package lists +from the server first:: + + root@39d693b2a75d:/sage# apt-get update + root@39d693b2a75d:/sage# apt-get install gcc + +Using Sage's database of distribution prerequisites +--------------------------------------------------- + +The source code of the Sage distribution contains a database of +package names in various distributions' package managers. For +example, the file ``build/pkgs/debian.txt`` contains the following + +.. code-block:: yaml + + # This file, build/pkgs/debian.txt, contains names of Debian/Ubuntu packages + # needed for installation of Sage from source. + # + # In addition, the files build/pkgs/SPKG/debian.txt contain the names + # of packages that provide the equivalent of SPKG. + # + # Everything on a line after a # character is ignored. + binutils + make + m4 + perl + # python3-minimal is not enough on debian buster, ubuntu bionic - it does not have urllib + python3 # system python for bootstrapping the build + tar + bc + gcc + # On debian buster, need C++ even to survive 'configure'. Otherwise: + # checking how to run the C++ preprocessor... /lib/cpp + # configure: error: in `/sage': + # configure: error: C++ preprocessor "/lib/cpp" fails sanity check + g++ + # Needed if we download some packages from a https upstream URL + ca-certificates + +From this information, we know that we can use the following command +on our container to install the necessary build prerequisites:: + + root@39d693b2a75d:/sage# apt-get install binutils make m4 perl python3 tar bc gcc g++ ca-certificates + Reading package lists... Done + Building dependency tree + Reading state information... Done + tar is already the newest version (1.29b-2ubuntu0.1). + The following additional packages will be installed: + ... + Done. + +(The Sage `Installation Guide <../installation/index.html>`_ also +provides such command lines for some distributions; these are +automatically generated from the database of package names.) + +Now we can start the build:: + + root@39d693b2a75d:/sage# ./configure + checking for a BSD-compatible install... /usr/bin/install -c + checking for root user... yes + configure: error: You cannot build Sage as root, switch to an unprivileged user. (If building in a container, use --enable-build-as-root.) + +Let's just follow this helpful hint:: + + root@39d693b2a75d:/sage# ./configure --enable-build-as-root + checking for a BSD-compatible install... /usr/bin/install -c + ... + + +Using Sage's database of equivalent distribution packages +--------------------------------------------------------- + +At the end of the ``./configure`` run, Sage issued a message like the +following:: + + configure: notice: the following SPKGs did not find equivalent system packages: arb boost boost_cropped bzip2 ... yasm zeromq zlib + checking for the package system in use... debian + configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: + configure: $ sudo apt-get install libflint-arb-dev ... yasm libzmq3-dev libz-dev + configure: After installation, re-run configure using: + configure: $ ./config.status --recheck && ./config.status + +This information comes from Sage's database of equivalent distribution +packages. For example:: + + root@39d693b2a75d:/sage# ls build/pkgs/arb/distros/ + arch.txt conda.txt debian.txt gentoo.txt + root@39d693b2a75d:/sage# cat build/pkgs/arb/distros/debian.txt + libflint-arb-dev + +Note that these package equivalencies are based on a current stable or +testing version of the distribution; the packages are not guaranteed +to exist in every release or derivative distribution. + +The Sage distribution is intended to build correctly no matter what +superset of the set of packages forming the minimal build +prerequisites is installed on the system. If it does not, this is a +bug of the Sage distribution and should be reported and fixed on a +ticket. Crucial part of a bug report is the configuration of the +system, in particular a list of installed packages and their versions. + +Let us install a subset of these packages:: + + root@39d693b2a75d:/sage# apt-get install libbz2-dev bzip2 yasm libz-dev + Reading package lists... Done + ... + Setting up zlib1g-dev:amd64 (1:1.2.11.dfsg-0ubuntu2) ... + root@39d693b2a75d:/sage# + + +Committing a container to disk +------------------------------ + +After terminating the container, we can create a new image +corresponding to its current state:: + + root@39d693b2a75d:/sage# ^D + [mkoeppe@sage worktree-ubuntu-latest]$ docker ps -a | head -n3 + CONTAINER ID IMAGE COMMAND CREATED STATUS + 39d693b2a75d ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 6 seconds ago + 9f3398da43c2 ubuntu:latest "/bin/bash" 8 minutes ago Exited (0) 8 minutes ago + [mkoeppe@sage worktree-ubuntu-latest]$ docker commit 39d693b2a75d ubuntu-latest-minimal-17 + sha256:4151c5ca4476660f6181cdb13923da8fe44082222b984c377fb4fd6cc05415c1 + +Here, ``39d693b2a75d`` was the container id (which appeared in the +shell prompts and in the output of ``docker ps``), and +``ubuntu-latest-minimal-17`` is an arbitrary symbolic name for the new +image. The output of the command is the id of the new image. We can +use either the symbolic name or the id to refer to the new image. + +We can run the image and get a new container with the same state as +the one that we terminated. Again we want to mount our worktree into +it; otherwise, because we did not make a copy, the new container will +have no access to the worktree:: + + [mkoeppe@sage worktree-ubuntu-latest]$ docker run -it \ + --mount type=bind,source=$(pwd),target=/sage ubuntu-latest-minimal-17 + root@73987568712c:/# cd sage + root@73987568712c:/sage# command -v gcc + /usr/bin/gcc + root@73987568712c:/sage# command -v yasm + /usr/bin/yasm + root@73987568712c:/sage# ^D + [mkoeppe@sage worktree-ubuntu-latest]$ + +The image ``ubuntu-latest-minimal-17`` can be run in as many +containers as we want and can also be shared with other users or +developers so that they can run it in a container on their machine. +(See the Docker documentation on how to `share images on Docker Hub +`_ or to `save images to a +tar archive +`_.) + +This facilitates collaboration on fixing portability bugs of the Sage +distribution. After reproducing a portability bug on a container, +several developers can work on fixing the bug using containers running +on their respective machines. + + +Generating Dockerfiles +---------------------- + +Sage also provides a script for generating a ``Dockerfile``, which is +a recipe for automatically building a new image:: + + [mkoeppe@sage sage]$ build/bin/write-dockerfile.sh debian "@(standard|optional)" > Dockerfile + +(The second argument is a bash extglob pattern for the package type.) + +.. this interface should be improved obviously. See #29146 - Refactor tox.ini and build/bin/write_dockerfile.sh + +The ``Dockerfile`` instructs the command ``docker build`` to build a +new Docker image. Let us take a quick look at the generated file; +this is slightly simplified:: + + [mkoeppe@sage sage]$ cat Dockerfile + # Automatically generated by SAGE_ROOT/build/bin/write-dockerfile.sh + # the :comments: separate the generated file into sections + # to simplify writing scripts that customize this file + ... + +First, it instructs ``docker build`` to start from an existing base +image...:: + + ... + ARG BASE_IMAGE=ubuntu:latest + FROM ${BASE_IMAGE} + ... + +Then, to install system packages...:: + + ... + RUN apt-get update && DEBIAN_FRONTEND=noninteractive apt-get install -qqq --no-install-recommends --yes binutils make m4 perl python3 ... yasm libzmq3-dev libz-dev && apt-get clean + +Then, to bootstrap and configure...:: + + RUN mkdir -p /sage + WORKDIR /sage + ADD Makefile VERSION.txt README.md bootstrap configure.ac sage ./ + ADD src/doc/bootstrap src/doc/bootstrap + ADD m4 ./m4 + ADD build ./build + ADD src/bin/sage-version.sh src/bin/sage-version.sh + RUN ./bootstrap + ADD src/bin src/bin + ADD src/Makefile.in src/Makefile.in + ARG EXTRA_CONFIGURE_ARGS="" + RUN ./configure --enable-build-as-root ${EXTRA_CONFIGURE_ARGS} || (cat config.log; exit 1) + +Finally, to build and test...:: + + ARG NUMPROC=8 + ENV MAKE="make -j${NUMPROC}" + ARG USE_MAKEFLAGS="-k" + RUN make ${USE_MAKEFLAGS} base-toolchain + ARG TARGETS_PRE="sagelib-build-deps" + RUN make ${USE_MAKEFLAGS} ${TARGETS_PRE} + ADD src src + ARG TARGETS="build ptest" + RUN make ${USE_MAKEFLAGS} ${TARGETS} + +You can customize the image build process by passing build arguments to the +command ``docker build``. For example:: + + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg BASE_IMAGE=ubuntu:latest \ + --build-arg NUMPROC=4 \ + --build-arg EXTRA_CONFIGURE_ARGS="--with-python=2" + +These arguments (and their default values) are defined using ``ARG`` +commands in the ``Dockerfile``. + +The above command will build Sage from scratch and will therefore take +quite long. Let us instead just do a partial build, consisting of one +small package, by setting the arguments ``TARGETS_PRE`` and +``TARGETS``. We use a silent build (``make V=0``):: + + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg TARGETS_PRE=ratpoints \ + --build-arg TARGETS=ratpoints \ + --build-arg USE_MAKEFLAGS="V=0" + Sending build context to Docker daemon 285MB + Step 1/28 : ARG BASE_IMAGE=ubuntu:latest + ... + Step 2/28 : FROM ${BASE_IMAGE} + ---> 549b9b86cb8d + ... + Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE} + ... + make[1]: Entering directory '/sage/build/make' + sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '/sage/logs/pkgs/ratpoints-2.1.3.p5.log' + [ratpoints-2.1.3.p5] installing. Log file: /sage/logs/pkgs/ratpoints-2.1.3.p5.log + [ratpoints-2.1.3.p5] successfully installed. + make[1]: Leaving directory '/sage/build/make' + + real 0m18.886s + user 0m1.779s + sys 0m0.314s + Sage build/upgrade complete! + ... + ---> 2d06689d39fa + Successfully built 2d06689d39fa + +We can now start a container using the image id shown in the last step:: + + [mkoeppe@sage sage]$ docker run -it 2d06689d39fa bash + root@fab59e09a641:/sage# ls -l logs/pkgs/ + total 236 + -rw-r--r-- 1 root root 231169 Mar 26 22:07 config.log + -rw-r--r-- 1 root root 6025 Mar 26 22:27 ratpoints-2.1.3.p5.log + root@fab59e09a641:/sage# ls -l local/lib/*rat* + -rw-r--r-- 1 root root 177256 Mar 26 22:27 local/lib/libratpoints.a + +You can customize the image build process further by editing the +``Dockerfile``. For example, by default, the generated ``Dockerfile`` +configures, builds, and tests Sage. By deleting or commenting out the +commands for the latter, you can adjust the Dockerfile to stop after +the ``configure`` phase, for example. + +``Dockerfile`` is the default filename for Dockerfiles. You can +change it to any other name, but it is recommended to use +``Dockerfile`` as a prefix, such as ``Dockerfile-debian-standard``. +It should be placed within the tree rooted at the current directory +(``.``); if you want to put it elsewhere, you need to learn about +details of "Docker build contexts". + +Note that in contrast to the workflow described in the above sections, +the ``Dockerfile`` **copies** a snapshot of your Sage worktree into +the build container, using ``ADD`` commands, instead of mounting the +directory into it. This copying is subject to the exclusions in the +``.gitignore`` file (via a symbolic link from ``.dockerignore``). +Therefore, only the sources are copied, but not your configuration +(such as the file ``config.status``), nor the ``$SAGE_LOCAL`` tree, +nor any other build artefacts. + +Because of this, you can build a Docker image using the generated +``Dockerfile`` from your main Sage development tree. It does not have +to be distclean to start, and the build will not write into it at all. +Hence, you can continue editing and compiling your Sage development +tree even while Docker builds are running. + + +Debugging a portability bug using Docker +---------------------------------------- + +Let us do another partial build. We choose a package that we suspect +might not work on all platforms, ``surf``, which was marked as +"experimental" in 2017:: + + [mkoeppe@sage sage]$ docker build . -f Dockerfile \ + --build-arg BASE_IMAGE=ubuntu:latest \ + --build-arg NUMPROC=4 \ + --build-arg TARGETS_PRE=surf \ + --build-arg TARGETS=surf + Sending build context to Docker daemon 285MB + Step 1/28 : ARG BASE_IMAGE=ubuntu:latest + Step 2/28 : FROM ${BASE_IMAGE} + ---> 549b9b86cb8d + ... + Step 24/28 : ARG TARGETS_PRE="sagelib-build-deps" + ---> Running in 17d0ddb5ad7b + Removing intermediate container 17d0ddb5ad7b + ---> 7b51411520c3 + Step 25/28 : RUN make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE} + ---> Running in 61833bea6a6d + make -j4 build/make/Makefile --stop + ... + [surf-1.0.6-gcc6] Attempting to download package surf-1.0.6-gcc6.tar.gz from mirrors + ... + [surf-1.0.6-gcc6] http://mirrors.mit.edu/sage/spkg/upstream/surf/surf-1.0.6-gcc6.tar.gz + ... + [surf-1.0.6-gcc6] Setting up build directory for surf-1.0.6-gcc6 + ... + [surf-1.0.6-gcc6] /usr/bin/ld: cannot find -lfl + [surf-1.0.6-gcc6] collect2: error: ld returned 1 exit status + [surf-1.0.6-gcc6] Makefile:504: recipe for target 'surf' failed + [surf-1.0.6-gcc6] make[3]: *** [surf] Error 1 + ... + [surf-1.0.6-gcc6] Error installing package surf-1.0.6-gcc6 + ... + Makefile:2088: recipe for target '/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6' failed + make[1]: *** [/sage/local/var/lib/sage/installed/surf-1.0.6-gcc6] Error 1 + make[1]: Target 'surf' not remade because of errors. + make[1]: Leaving directory '/sage/build/make' + ... + Error building Sage. + + The following package(s) may have failed to build (not necessarily + during this run of 'make surf'): + + * package: surf-1.0.6-gcc6 + last build time: Mar 26 22:07 + log file: /sage/logs/pkgs/surf-1.0.6-gcc6.log + build directory: /sage/local/var/tmp/sage/build/surf-1.0.6-gcc6 + + ... + Makefile:31: recipe for target 'surf' failed + make: *** [surf] Error 1 + The command '/bin/sh -c make SAGE_SPKG="sage-spkg -y -o" ${USE_MAKEFLAGS} ${TARGETS_PRE}' returned a non-zero code: 2 + +Note that no image id is shown at the end; the build failed, and no +image is created. However, the container in which the last step of +the build was attempted exists:: + + [mkoeppe@sage sage]$ docker ps -a |head -n3 + CONTAINER ID IMAGE COMMAND CREATED STATUS + 61833bea6a6d 7b51411520c3 "/bin/sh -c 'make SA…" 9 minutes ago Exited (2) 1 minute ago + 73987568712c ubuntu-latest-minimal-17 "/bin/bash" 24 hours ago Exited (0) 23 hours ago + +We can copy the build directory from the container for inspection:: + + [mkoeppe@sage sage]$ docker cp 61833bea6a6d:/sage/local/var/tmp/sage/build ubuntu-build + [mkoeppe@sage sage]$ ls ubuntu-build/surf*/src + AUTHORS TODO curve misc + COPYING acinclude.m4 debug missing + ChangeLog aclocal.m4 dither mkinstalldirs + INSTALL background.pic docs mt + Makefile config.guess draw src + Makefile.am config.log drawfunc surf.1 + Makefile.global config.status examples surf.xpm + Makefile.in config.sub gtkgui yaccsrc + NEWS configure image-formats + README configure.in install-sh + +Alternatively, we can use ``docker commit`` as explained earlier to +create an image from the container:: + + [mkoeppe@sage sage]$ docker commit 61833bea6a6d + sha256:003fbd511016fe305bd8494bb1747f0fbf4cb2c788b4e755e9099d9f2014a60d + [mkoeppe@sage sage]$ docker run -it 003fbd511 bash + root@2d9ac65f4572:/sage# (cd /sage/local/var/tmp/sage/build/surf* && /sage/sage --buildsh) + + Starting subshell with Sage environment variables set. Don't forget + to exit when you are done. + ... + Note: SAGE_ROOT=/sage + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ls /usr/lib/libfl* + /usr/lib/libflint-2.5.2.so /usr/lib/libflint-2.5.2.so.13.5.2 /usr/lib/libflint.a /usr/lib/libflint.so + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get update && apt-get install apt-file + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file update + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-file search "/usr/lib/libfl.a" + flex-old: /usr/lib/libfl.a + freebsd-buildutils: /usr/lib/libfl.a + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ apt-get install flex-old + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ ./spkg-install + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + /usr/bin/install -c surf /sage/local/bin/surf + /usr/bin/install -c -m 644 ./surf.1 /sage/local/share/man/man1/surf.1 + ... + make[1]: Leaving directory '/sage/local/var/tmp/sage/build/surf-1.0.6-gcc6/src' + (sage-buildsh) root@2d9ac65f4572:surf-1.0.6-gcc6$ exit + root@2d9ac65f4572:/sage# exit + [mkoeppe@sage sage]$ + +A standard case of bitrot. + + +Automatic Docker-based build testing using tox +---------------------------------------------- + +`tox `_ is a Python package that +is widely used for automating tests of Python projects. + +Install ``tox`` for use with your system Python, for example using:: + + [mkoeppe@sage sage]$ pip install --user tox + +A tox "environment" is identified by a symbolic name composed of +several `Tox "factors" +`_, +which are defined in the file ``$SAGE_ROOT/tox.ini``. + +The **technology** factor describes how the environment is run: + +- ``docker`` builds a Docker image as described above. + +- ``local`` runs testing on the host OS instead. We explain this + technology in a later section. + +The next two factors determine the host system configuration: The +**system factor** describes a base operating system image. + +- Examples are ``ubuntu-focal``, ``debian-buster``, + ``archlinux-latest``, ``fedora-30``, ``slackware-14.2``, + ``centos-7-i386``, and ``ubuntu-bionic-arm64``. + +- See ``$SAGE_ROOT/tox.ini`` for a complete list, and to which images + on Docker hub they correspond. + +The **packages factor** describes a list of system packages to be +installed on the system before building Sage: + +- ``minimal`` installs the system packages known to Sage to provide + minimal prerequisites for bootstrapping and building the Sage + distribution. + +- ``standard`` additionally installs all known system packages that + are equivalent to standard packages of the Sage distribution, for + which the mechanism ``spkg-configure.m4`` is implemented. + This corresponds to the type pattern ``@(standard)``. + +- ``maximal`` does the same for all standard and optional packages. + This corresponds to the type pattern ``@(standard|optional)``. + +The factors are connected by a hyphen to name a system configuration, +such as ``debian-buster-standard`` and ``centos-7-i386-minimal``. + +Finally, the **configuration** factor (which is allowed to be empty) +controls how the ``configure`` script is run. + +- ``python2`` adds the argument ``--with-python=2`` to the + ``configure`` run. + +The factors are connected by a hyphen to name a tox environment. (The +order of the factors does not matter; however, for consistency and +because the ordered name is used for caching purposes, we recommend to +use the factors in the listed order.) + +To run an environment:: + + [mkoeppe@sage sage]$ tox -e docker-slackware-14.2-minimal + [mkoeppe@sage sage]$ tox -e docker-ubuntu-bionic-standard-python2 + +Extra arguments to ``docker build`` can be supplied through the +environment variable ``EXTRA_DOCKER_BUILD_ARGS``. For example, for +a silent build (``make V=0``), use:: + + [mkoeppe@sage sage]$ EXTRA_DOCKER_BUILD_ARGS="--build-arg USE_MAKEFLAGS=\"V=0\"" \ + tox -e docker-ubuntu-bionic-standard + +Automatic build testing on the host OS using tox -e local-direct +---------------------------------------------------------------- + +The ``local`` technology runs testing on the host OS instead. + +In contrast to the ``docker`` technology, it does not make a copy of +the source tree. It is most straightforward to run it from a +separate, distclean git worktree. + +Let us try a first variant of the ``local`` technology, the tox +environment called ``local-direct``. Because all builds with tox +begin by bootstrapping the source tree, you will need autotools and +other prerequisites installed in your system. See +``build/pkgs/*-bootstrap.txt`` for a list of system packages that +provide these prerequisites. + +We start by creating a fresh (distclean) git worktree. + + [mkoeppe@sage sage] git worktree add worktree-local + [mkoeppe@sage sage] cd worktree-local + [mkoeppe@sage worktree-local] ls + COPYING.txt ... Makefile ... configure.ac ... src tox.ini + +Again we build only a small package. Build targets can be passed as +positional arguments (separated from tox options by ``--``):: + + [mkoeppe@sage worktree-local] tox -e local-direct -- ratpoints + local-direct create: /Users/mkoeppe/.../worktree-local/.tox/local-direct + local-direct run-test-pre: PYTHONHASHSEED='2211987514' + ... + src/doc/bootstrap:48: installing src/doc/en/installation/debian.txt... + bootstrap:69: installing 'config/config.rpath' + configure.ac:328: installing 'config/compile' + configure.ac:113: installing 'config/config.guess' + ... + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + sage-logger -p 'sage-spkg -y -o ratpoints-2.1.3.p5' '.../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log' + [ratpoints-2.1.3.p5] installing. Log file: .../worktree-local/logs/pkgs/ratpoints-2.1.3.p5.log + [ratpoints-2.1.3.p5] successfully installed. + ... + local-direct: commands succeeded + congratulations :) + +Let's investigate what happened here:: + + [mkoeppe@sage worktree-local]$ ls -la + total 2576 + drwxr-xr-x 35 mkoeppe staff 1120 Mar 26 22:20 . + drwxr-xr-x 63 mkoeppe staff 2016 Mar 27 09:35 .. + ... + lrwxr-xr-x 1 mkoeppe staff 10 Mar 26 20:34 .dockerignore -> .gitignore + -rw-r--r-- 1 mkoeppe staff 74 Mar 26 20:34 .git + ... + -rw-r--r-- 1 mkoeppe staff 1212 Mar 26 20:41 .gitignore + ... + drwxr-xr-x 7 mkoeppe staff 224 Mar 26 22:11 .tox + ... + -rw-r--r-- 1 mkoeppe staff 7542 Mar 26 20:41 Makefile + ... + lrwxr-xr-x 1 mkoeppe staff 114 Mar 26 20:45 config.log -> .tox/local-direct/log/config.log + -rwxr-xr-x 1 mkoeppe staff 90411 Mar 26 20:46 config.status + -rwxr-xr-x 1 mkoeppe staff 887180 Mar 26 20:45 configure + -rw-r--r-- 1 mkoeppe staff 17070 Mar 26 20:41 configure.ac + ... + lrwxr-xr-x 1 mkoeppe staff 103 Mar 26 20:45 logs -> .tox/local-direct/log + drwxr-xr-x 24 mkoeppe staff 768 Mar 26 20:45 m4 + lrwxr-xr-x 1 mkoeppe staff 105 Mar 26 20:45 prefix -> .tox/local-direct/local + -rwxr-xr-x 1 mkoeppe staff 4868 Mar 26 20:34 sage + drwxr-xr-x 16 mkoeppe staff 512 Mar 26 20:46 src + -rw-r--r-- 1 mkoeppe staff 13478 Mar 26 20:41 tox.ini + drwxr-xr-x 4 mkoeppe staff 128 Mar 26 20:46 upstream + +There is no ``local`` subdirectory. This is part of a strategy to +keep the source tree clean to the extent possible. In particular: + +- ``tox`` configured the build to use a separate ``$SAGE_LOCAL`` + hierarchy in a directory under the tox environment directory + ``.tox/local-direct``. It created a symbolic link ``prefix`` that + points there, for convenience:: + + [mkoeppe@sage worktree-local]$ ls -l prefix/lib/*rat* + -rw-r--r-- 1 mkoeppe staff 165968 Mar 26 20:46 prefix/lib/libratpoints.a + +- Likewise, it created a separate ``logs`` directory, again under the + tox environment directory, and a symbolic link. + +This makes it possible for advanced users to test several ``local`` +tox environments (such as ``local-direct`` and +``local-direct-python2``) out of one worktree. However, because a +build still writes configuration scripts and build artefacts (such as +``config.status``) into the worktree, only one ``local`` build can run +at a time in a given worktree. + +The tox environment directory will be reused for the next ``tox`` run, +which will therefore do an incremental build. To start a fresh build, +you can use the ``-r`` option. + +Automatic build testing on the host OS with best-effort isolation using tox -e local +------------------------------------------------------------------------------------ + +``tox -e local`` (without ``-direct``) attempts a best-effort +isolation from the user's environment as follows: + +- All environment variables are set to standard values; with the + exception of ``MAKE`` and ``EXTRA_CONFIGURE_ARGS``. In particular, + ``PATH`` is set to just ``/usr/bin:/bin:/usr/sbin:/sbin``; it does + not include ``/usr/local/bin``. + + +Note, however, that various packages have build scripts that use +``/usr/local`` or other popular file system locations such as +``/opt/sfw/``. Therefore, the isolation is not complete. Using +``/usr/local`` is considered standard behavior. On the other hand, we +consider a package build script that inspects other file system +locations to be a bug of the Sage distribution, which should be +reported and fixed on a ticket. + + +Automatic build testing on macOS with a best-effort isolated installation of Homebrew +------------------------------------------------------------------------------------- + +XCode on macOS does not provide the prerequisites for bootstrapping +the Sage distribution. A good way to install them is using the +Homebrew package manager. + +In fact, Sage provides a tox environment that automatically installs +an isolated copy of Homebrew with all prerequisites for bootstrapping:: + + [mkoeppe@sage worktree-local]$ tox -e local-homebrew-macos-minimal -- lrslib + local-homebrew-macos-minimal create: .../worktree-local/.tox/local-homebrew-macos-minimal + local-homebrew-macos-minimal run-test-pre: PYTHONHASHSEED='4246149402' + ... + Initialized empty Git repository in .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/.git/ + ... + Tapped 2 commands and 4942 formulae (5,205 files, 310.7MB). + ==> Downloading https://ftp.gnu.org/gnu/gettext/gettext-0.20.1.tar.xz + ... + ==> Pouring autoconf-2.69.catalina.bottle.4.tar.gz + ... + ==> Pouring pkg-config-0.29.2.catalina.bottle.1.tar.gz + .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/Cellar/pkg-config/0.29.2: 11 files, 623.4KB + ==> Caveats + ==> gettext + gettext is keg-only, which means it was not symlinked into .../worktree-local/.tox/local-homebrew-macos-minimal/homebrew, + because macOS provides the BSD gettext library & some software gets confused if both are in the library path. + + If you need to have gettext first in your PATH run: + echo 'export PATH=".../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/bin:$PATH"' >> ~/.bash_profile + + For compilers to find gettext you may need to set: + export LDFLAGS="-L.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/lib" + export CPPFLAGS="-I.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/opt/gettext/include" + ... + local-homebrew-macos-minimal run-test: commands[0] | bash -c 'export PATH=.../worktree-local/.tox/local-homebrew-macos-minimal/homebrew/bin:/usr/bin:/bin:/usr/sbin:/sbin && . .homebrew-build-env && ./bootstrap && ./configure --prefix=.../worktree-local/.tox/local-homebrew-macos-minimal/local && make -k V=0 ... lrslib' + ... + bootstrap:69: installing 'config/config.rpath' + ... + checking for a BSD-compatible install... /usr/bin/install -c + checking whether build environment is sane... yes + ... + configure: notice: the following SPKGs did not find equivalent system packages: arb cbc cliquer ... tachyon xz yasm zeromq + checking for the package system in use... homebrew + configure: hint: installing the following system packages is recommended and may avoid building some of the above SPKGs from source: + configure: $ brew install cmake gcc gsl mpfi ninja openblas gpatch r readline xz yasm zeromq + ... + sage-logger -p 'sage-spkg -y -o lrslib-062+autotools-2017-03-03.p1' '.../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log' + [lrslib-062+autotools-2017-03-03.p1] installing. Log file: .../worktree-local/logs/pkgs/lrslib-062+autotools-2017-03-03.p1.log + [lrslib-062+autotools-2017-03-03.p1] successfully installed. + ... + local-homebrew-macos-minimal: commands succeeded + congratulations :) + +The tox environment uses the subdirectory ``homebrew`` of the +environment directory ``.tox/local-homebrew-macos-minimal`` as the +Homebrew prefix. This installation does not interact in any way with +a Homebrew installation in ``/usr/local`` that you may have. + +The test script sets the ``PATH`` to the ``bin`` directory of the +Homebrew prefix, followed by ``/usr/bin:/bin:/usr/sbin:/sbin``. It +then uses the script ``$SAGE_ROOT/.homebrew-build-env`` to set +environment variables so that Sage's build scripts will find +"keg-only" packages such as ``gettext``. + +The ``local-homebrew-macos-standard`` environment additionally +installs (in its separate isolated copy of Homebrew) all Homebrew +packages known to Sage for which the ``spkg-configure.m4`` mechanism +is implemented; this is similar to the ``docker-standard`` tox +environments described earlier. + + +Automatic parallel tox runs on GitHub Actions +--------------------------------------------- + +The Sage source tree includes a default configuration for GitHub +Actions that runs tox on a multitude of platforms on every pull +request to a repository for which GitHub Actions are enabled. + +This is defined in the file ``$SAGE_ROOT/.github/workflows/tox.yml``. + +An additional GitHub Actions workflow for testing on Cygwin, not based +on tox, is defined in the file +``$SAGE_ROOT/.github/workflows/ci-cygwin.yml``. + +GitHub Actions runs these build jobs on 2-core machines with 7 GB of +RAM memory and 14 GB of SSD disk space, cf. +`here `_, +and has a time limit of 6h per job. This is just barely enough for a +typical ``minimal`` build followed by ``make ptest`` to succeed; and +plenty of time for a typical ``standard`` build to succeed. + +Build logs become available as "artifacts" when all jobs of the +workflow have finished. Each job generates one tarball. +"Annotations" highlight certain top-level errors or warnings issued +during the build. + +The following procedure seems to work well for testing branches during +development: + +- Create a branch from a recent beta release that contains the default + GitHub Actions configuration; name it ``TESTER``, say. + +- Edit ``$SAGE_ROOT/.github/workflows/tox.yml`` to include the system + config wish to test. + +- Commit and push the branch to your GitHub fork of sage. + +- Push your development branch to your GitHub repository and create a + pull request against the ``TESTER`` branch. This will trigger the + GitHub Actions workflow. +