Enable VPATH builds (several independent build trees connected to one source tree)

[mkoeppe]

Executive summary Currently, --prefix selects the install tree (SAGE_LOCAL) of sage-the-distribution. But often we want several builds out of (almost) the same source tree.

The vpath mechanism proposed here selects the build tree (SAGE_ROOT) of sage-the-distribution. This includes the configuration and build artefacts such as src/build/....

---

VPATH builds are a powerful feature of autotools (and other) build systems. This feature allows the developer to build a package in a separate (initially empty) build directory tree. The source directory tree is only read from (and could even be mounted read-only) and is therefore always clean.

This is a powerful feature for the developer because from the same source tree many different configurations can be built and tested, without having to go through "make distclean" and reconfiguration. The source tree can also be shared, for example using a networked file system between different hosts, running different architectures. Another modern use case involves VMs. For example, Docker allows to mount the source directory from the host in the VM (see #21474).

For Sage, "different configurations" could mean different architectures (via VMs), different sets of installed packages, Py2 vs Py3, etc.

This is how it is used.

 cd SAGE_ROOT
 autoreconf

and then

 mkdir BUILDDIR
 cd BUILDDIR
 SRCDIR/configure --srcdir=SRCDIR
 make

The present patch implements this. BUILDDIR from the above example is reflected in the old environment variable $SAGE_ROOT; this is now to be distinguished from SRCDIR, which is reflected in the new environment variable $SAGE_VPATH. $SAGE_LOCAL defaults to BUILDDIR/local, but can of course be changed using configure --prefix.

The present patch changes the various places where $SAGE_ROOT is used.
When "./configure" detects a VPATH build, it installs patched copies of sage, Makefile and build/make/install in BUILDDIR.

With the current set of patches, the compilation goes through and gives a working sage, except for the docbuild.

CC: @sagetrac-felixs @jdemeyer @kiwifb @embray @nexttime @vbraun @dimpase @fchapoton @jhpalmieri

Component: build: configure

Work Issues: merge conflicts

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/vpath_build @ 05f3a91

Issue created by migration from https://trac.sagemath.org/ticket/21469

[mkoeppe]

Branch: u/mkoeppe/vpath_build

[mkoeppe]

Commit: ac27cdc

[mkoeppe]

New commits:

ac27cdc

configure.ac: VPATH fixes

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

5698f8e	Distinguish SAGE_ROOT and SAGE_SRC_ROOT for VPATH build
a98cfa2	Use AC_CONFIG_COMMANDS
e9b1c0f	Symlink build/pkgs (temporary solution)

[sagetrac-git]

Changed commit from ac27cdc to e9b1c0f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

e15ada6

Define SAGE_SRC_ROOT for VPATH build

[sagetrac-git]

Changed commit from e9b1c0f to e15ada6

[sagetrac-git]

Changed commit from e15ada6 to 9323c01

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

9323c01

Use SAGE_SRC_ROOT in package scripts

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

7a3913a

More SAGE_SRC_ROOT

[sagetrac-git]

Changed commit from 9323c01 to 7a3913a

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

4cc3954

More SAGE_SRC_ROOT

[sagetrac-git]

Changed commit from 7a3913a to 4cc3954

[mkoeppe]

comment:9

The current patch makes some high-level preparations for a VPATH build, in particular the package system.

"make" runs through until it gets to sagelib, which needs some more VPATH work (the current patch tries to get away by just making a symlink from $SAGE_SRC_ROOT/src to $SAGE_ROOT/src, which of course fails).

I've set this to 'needs review' in the hope of receiving some feedback regarding the approach of this patch.

[kiwifb]

comment:10

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

This is most evident in the latest commit. As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Other people may have a different opinion but there is no value in this for me so far. Possibly more work in fact.

[mkoeppe]

comment:11

Replying to @kiwifb:

I am not sure what the objective of this ticket is. Considering the nature of sage's build system I am not sure what VPATH brings me.

A VPATH build makes it easy to build from the same source in many different configurations. Concretely, I've started to run some VMs (with Linux 64 bit, 32 bit) via Docker, which mount the Sage source directory from the host.

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

7bb8718

Make sage script in build dir executable

[sagetrac-git]

Changed commit from 4cc3954 to 7bb8718

[mkoeppe]

comment:13

Replying to @kiwifb:

As a distro maintainer of sage there are already build time things in env.py that are not needed at runtime and in a way are superfluous. You have just added one. Here the problem that I am forced to consider, is that going by the separation rules that I try to follow, there should be stuff in sage/env.py and some in a new file sage_setup/build_env.py but that's marginal to what you are doing.

Yes, this ticket does not address any of the issues that would arise in distribution packaging.

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

[kiwifb]

comment:14

I should apologize for being so negative. Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

[mkoeppe]

comment:15

Replying to @kiwifb:

I should apologize for being so negative.

No worries.

Currently configure is run by the initial call to make, it would be nice to get to the stage where we run configure first :)

That already works with this patch.

Seriously, yes VPATH as lots of advantages, since most people run sage from where they build it, being able to do several builds from the same source without make distclean or git clone is a good thing.

I guess, it mostly started my brain on the fact that the separation build and runtime is not clean in vanilla sage. Your VPATH work may help to clarify it in some areas. Let's try to minimise where it would mix things further.

Sounds good. Yes, I'm hoping too that it helps clarify some things as a side-effect.

[jdemeyer]

comment:16

Replying to @mkoeppe:

Instead it adds a standard feature expected from autotools build system, for those who use sage as a distribution.

I sort of agree with François.

Part of the problem is that Sage is a distribution (a collection of packages) as well as a Python library. The fact this this is currently mixed up in the sources (the sources of the distribution-part and the library-part are in one repo) makes it more difficult to understand what this ticket is about.

For the distribution part, there really is no such thing as a build directory. There is a a temporary directory in local/var/tmp/sage/build but I would not consider that a build directory in the usual sense.

For the library part, a VPATH build is meaningful. Although, I think that VPATH builds are unusual in the Python world. Luckily, the default build directories are platform-dependendent. Anyway, it seems like this ticket isn't about that.

[jdemeyer]

comment:17

I think the ticket description should have a lot more information about

1. which problem that this ticket is trying to solve (the why).

2. what the branch actually does (the what).

Note that this are 2 independent things (which are unfortunately often confused).

[mkoeppe]

comment:111

Setting new milestone based on a cursory review of ticket status, priority, and last modification date.

[mkoeppe]

comment:112

Setting a new milestone for this ticket based on a cursory review.