Long-term task ticket: Support "make install" of the Sage distribution

[mkoeppe]

#21479 makes SAGE_LOCAL configurable by "./configure --prefix=SAGE_LOCAL". The default is the "local" subdirectory of the sage source tree (or the separate sage build tree, if using a VPATH configuration - #21469).

On this task ticket, let's organize the steps necessary to support "make install" that is separate from "make".

• Step 1: Autotoolize ALL packages, so that they support staged installs using DESTDIR
• Step 2: ...
• Step 3: Have packages install into staging hierarchy and have make install copy the staged stuff to the prefix hierarchy. Easy! ;)

The comments section of this ticket is also a place for philosophical discussions regarding the nature of build and install directories, etc.

CC: @dimpase @sagetrac-felixs @jdemeyer @mezzarobba

Component: build

Issue created by migration from https://trac.sagemath.org/ticket/21495

[sagetrac-felixs]

comment:1

step 1 Autotoolize ALL packages

wow. that's a tremendous effort. i don't think it's really necessary.

but even if: why would the next step be "staged installs"? an autotoolized package will work fine after just "make". in contrast, it will not necessarily be functional from within $DESTDIR, that's not the intent of DESTDIR.

the package transition from "install to SAGE_LOCAL (during build)" + "unable to install to $prefix" to "dont install during build" + "possibly install to $prefix" can be done one after another. by someone who needs the package installed by sage. for this to make sense, it must be possible to disable packages... that's implemented (drafted) in #14796.

think of "patch". as long as nobody wants to install sage on a platform without patch, it will not be necessary to have an installable patch package provided by sage. sage will still work on this platform the traditional way, with patch "installed" to $SAGE_LOCAL, as usual ($PATH will take care of it).

my sage-autotools draft also used "staging" for "prepare something that can be used right now before installation". the fallback/transition was "install to SAGE_LOCAL". back then, i only needed ~3 packages installable by sage. the rest was already there (hence "disabled" at top level).

i think step 0 is missing. imo it is "implement configure for sagelib". without configure it will be harder to pass things like --with-$program=$somewhere. also sagelib might need (environment?) overrides for several paths.

how does "disable packages at top level" sound to you?

[jdemeyer]

Replying to @mkoeppe:

On this task ticket, let's organize the steps necessary to support "make install" that is separate from "make".

What is the point of that?

[mkoeppe]

comment:3

Mostly I want 1 central place where this is discussed to keep productive tickets clear of this discussion.

[jdemeyer]

comment:4

OK, for the record: I completely disagree with this ticket.

[jdemeyer]

comment:5

Replying to @dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else.

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense. I cannot easily install stuff from /usr (say) to another directory and that's not an issue. If you think of $SAGE_LOCAL as the installation directory, analogously to /usr, then I really do not see the point.

Of course, I agree that we should support #21479. Maybe if the installation directory becomes configurable, people will start understanding that it really is an installation directory.

[sagetrac-felixs]

comment:6

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense.

exactly. SAGE_LOCAL does not make sense. it is a (historic) workaround, and an implementation detail. not a requirement.

Of course, I agree that we should support #21479.

#21479 is ./configure --prefix=SAGE_LOCAL. what should (in your opinion) make (not make install) do in

    $ ./configure --prefix=SAGE_LOCAL
    $ make

?

you know i think the answer is "what every other build system would do": build stuff. no other build system requires a writable $prefix for what it does.

[jdemeyer]

comment:7

Replying to @sagetrac-felixs:

you know i think the answer is "what every other build system would do": build stuff. no other build system requires a writable $prefix for what it does.

But Sage-the-distribution is not a program, it is a distribution. It doesn't really make sense to "build" Sage-the-distribution. A separate build/install step really only makes sense for individual packages.

So the top-level make command installs Sage-the-distribution in its installation directory, which is $SAGE_LOCAL.

[dimpase]

comment:8

Replying to @jdemeyer:

Replying to @dimpase:

$SAGE_LOCAL is fine as long as you can also install things from there somewhere else.

You want to install stuff from $SAGE_LOCAL to another place? That does not really make sense. I cannot easily install stuff from /usr (say) to another directory and that's not an issue.

Surely, you normally never install stuff from /usr anywhere, because /usr is an installation place, not a place to build stuff.

E.g. /usr can be NFS-mounted and used by lots of users, and thus unsuitable for build work.

And BEFORE installing stuff, you want to make check.

[sagetrac-felixs]

comment:9

But Sage-the-distribution is not a program

it's a software package. and it's not the first software package with subpackages. it's just working differently (for historical reasons).

question is how to escape this dilemma. answer is: apply standard practices/terminology.

in the meantime, the transition should better not affect any of the users and developers.

i don't really grasp why you are against progress.

[jdemeyer]

comment:10

Replying to @dimpase:

Surely, you normally never install stuff from /usr anywhere, because /usr is an installation place, not a place to build stuff.

Right, just like $SAGE_LOCAL.

[jdemeyer]

comment:11

Replying to @sagetrac-felixs:

i don't really grasp why you are against progress.

I am not against progress. I am against pointless changes, especially if those changes make stuff more complicated without any advantage.

[jdemeyer]

comment:12

Replying to @sagetrac-felixs:

But Sage-the-distribution is not a program

it's a software package. and it's not the first software package with subpackages.

I do not see it as one package with subpackages. It is a collection of packages, installed in a common prefix.

[sagetrac-felixs]

comment:13

Replying to @jdemeyer:

more complicated without any advantage.

i did not say that enabling users to install sage is meant to be an advantage for you in particular.
my impression is, you are a developer not only a user.

look at the packages that you use, without being a developer of that particular package. sure i don't know what they are, statistically it's stuff like an OS kernel, a web browser, a desktop environment, a mail client, some office-suite... endless list.

would you say that it's not an advantage to have all these as packages ready in a zillon of distros that you could choose from?

(this is again drifting towards being off-topic. maybe it should be relacated to sage-flame)

[sagetrac-felixs]

comment:14

Replying to @jdemeyer:

I do not see it as one package with subpackages. It is a collection of packages, installed in a common prefix.

what you describe is stuff like "pip" or some "ebuild" package manager, or maybe "apt" (together with the corresponding package directories)

sage-the-distribution has its place, but it's nowhere close. in architecure and use it closely resembles a standard package. it's just lacking some functionality

[embray]

comment:16

I think I wrote elsewhere before I saw that this ticket already exists, but I'll repeat the basic idea here:

I think that in order to support this functionality, what we ought to do is break up all existing spkg-installs into separate spkg-build and spkg-install scripts--possibly also an spkg-env script specifically for setting environment variables needed for that package at both "build" and "install" time.

This would also ease further elimination of duplication between spkg-installs. For example, most packages' spkg-install becomes little more than a symlink to a stub script that calls make install.

Some packages can't easily be broken into cleanly separate "build" and "install" steps. I suppose this is the main reason for point 1 in the ticket description. I think "autotoolize" all packages is a big undertaking though, and not even appropriate for truly "all" packages (even excluding Python ones). For those packages that can't be separated, having a single "install" script is probably about as good as we'll get.

Where Python packages are concerned, ironically before switching to pip there was a clearer separation--you could ./setup.py build and ./setup.py install. For most pure-Python packages there's no reason to make this distinction, but for packages with extension modules, of which we use many, one could see some small argument for a separate build phase. A simple workaround for this with pip is to do pip install into an alternative --target (wherever the VPATH builds are going, for example). Then the install phase is just a matter of copying the built install into the appropriate site-packages for the Python being used to run Sage (which may even be the system Python--the ultimate goal).

[embray]

comment:17

(and to be clear, I agree with felix that "autotoolize all packages", even if you exclude the obvious ones where it doesn't apply, is probably more work than necessary--would be nice to do where it is easy, but I think simply separating build and install steps in the spkgs is an easier way to go)

[jdemeyer]

comment:18

Replying to @embray:

separating build and install steps in the spkgs is an easier way to go

What do we gain from that? I am totally missing that from the ticket description. Usually, there is a problem that you want to solve and then you look for a solution. So, what exactly is the problem that we are trying to solve here?

[embray]

comment:19

It would actually be great if I could test building Sage along with sagelib without having to actually muck with or break things in my existing Sage "installation" (i.e. currently wherever $SAGE_LOCAL is) that I'm using to run and test development just on sagelib. It would also be great for testing if make <packagename> just tried to build it but didn't install it anywhere.

Currently the only alternatives are to constantly muck with environment variables (and inevitably make mistakes) or have about a dozen simultaneous sage source dirs as it seems I currently do...

[mkoeppe]

comment:20

I strongly recommend not to spend time on this ticket. It is here to centralize discussion that will invariably come up.

There are plenty of build tickets which need actual work or review.

[embray]

comment:21

Replying to @mkoeppe:

I strongly recommend not to spend time on this ticket. It is here to centralize discussion that will invariably come up.

Right now you mean ;)

There are plenty of build tickets which need actual work or review.

Agreed--there's lots else to be done before this should be worked on at all,, though I'm all for having the discussion.

[embray]

comment:22

One obvious problem with this that I'm surprised hasn't been pointed out yet (unless it has, and I'm missing it--or at least, it hasn't been pointed out in this ticket) is that so many packages in Sage require their dependencies to be installed in order to build. So you can't do a make that builds all the packages in the Sage distribution, and then a separate make install to install them since most packages aren't going to build in the first place until and unless their dependencies have been installed.

This could be worked around by looking for the dependencies in their temporary build roots, but that could also get tricky fast. The only way this can be made to work is if every package were autotoolized, but that's not going to happen and is beyond the scope of what Sage's packaging system does.

I think the work I'm doing now for #22509 will ultimately satisfy my needs for easily building packages without installing them. Right now the install step is still run unconditionally but it will be easy to add a "build only" flag in the appropriate places once this work is done.