./bootstrap: Format build/pkgs/SPKG/distros/ information to produce the apt-get/yum command lines shown in the installation manual

[embray]

When trying to build Sage on a very bare fedora install (e.g. a Docker container) the packages listed in the build instructions at:

https://doc.sagemath.org/html/en/installation/source.html#linux-prerequisite-installation

are insufficient. At the very least it also needs findutils and which, and currently it also requires python2, though that should be fixed as part of #26953. The need for which comes, at the very least, from MPIR's configure script, which uses which at least on Linux.

With #29053, the necessary distribution packages for an installation are cataloged in build/pkgs/fedora.txt, build/pkgs/SPKG/distros/fedora.txt and similar for other distributions.

In this ticket, we add to bootstrap some code to keep the yum (and apt-get for debian) command-lines in the installation manual up to date. #29053 prepared this by isolating these command lines in separate .txt files in src/doc.

CC: @dimpase @embray @jhpalmieri

Component: documentation

Author: Matthias Koeppe

Branch: 688c68e

Reviewer: John Palmieri, Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/26964

[embray]

comment:2

Retarging tickets optimistically to the next milestone. If you are responsible for this ticket (either its reporter or owner) and don't believe you are likely to complete this ticket before the next release (8.7) please retarget this ticket's milestone to sage-pending or sage-wishlist.

[embray]

comment:3

Removing most of the rest of my open tickets out of the 8.7 milestone, which should be closed.

[mkoeppe]

comment:4

Added these 2 packages to #29053.

[mkoeppe]

Dependencies: #29053

[mkoeppe]

Changed dependencies from #29053 to none

[embray]

comment:6

I'm not sure how I feel about using the bootstrap script for this purpose. I think I would prefer if it were done by configure instead. Reason: The bootstrap script should really be limited to tasks specific to generating the configure file (whether running autoreconf or downloadng it). Other than that, I think this is a great idea!

[mkoeppe]

comment:7

No, no, bootstrap is exactly the right phase for this, not configure. Let me explain more.

bootstrap will prepare a source tree so that

• make sdist can be run
• configure can be run by a user (who does not have autotools)
• src/setup.py sdist can be run.
• src/configure can be run (Add an autoconf configure script for sagelib #29119)

None of this depends on the configuration determined by configure!

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

[embray]

comment:8

Replying to @mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

[embray]

comment:9

Couldn't this also be done as part of make sdist?

[mkoeppe]

comment:10

Replying to @embray:

Replying to @mkoeppe:

As the installation manual sources are part of the sagelib sdist, the command lines of this ticket should be prepared in the bootstrap phase.

I want to clarify something about this here: Do you mean the sagelib sdist, or the sage-the-distribution sdist?

Both.

Currently, of course, only sage-the-distribution has a working sdist.

sagelib's sdist needs fixing. You might remember ticket #21516 - Fix sagelib sdist (src/setup.py sdist).

I sort of see your point, but I'm still not sure why it needs to go here. Also, as part of #29119 I'm adding a src/bootstrap script for sagelib itself, so maybe something like this would make more sense to go there.

In my opinion, there should only be one developer-invoked bootstrap script (the one for the distribution) -- because everything will depend on the distribution's build/pkgs files.

SAGE_ROOT/bootstrap could certainly call a new script src/bootstrap for generating files that end up in src/. If you create that in #29119, fine.

Anyways, setting back to needs_work since no branch is attached. Maybe then I'll see what you're talking about.

Sure. #29041 (src/requirements.txt, src/constraints.txt, src/setup.cfg) has a branch, needs review.

[mkoeppe]

comment:11

Replying to @embray:

Couldn't this also be done as part of make sdist?

No, because developers who build sage out of a git worktree do not run make sdist.

[mkoeppe]

Branch: u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual

[sagetrac-git]

Commit: 438bc7f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

438bc7f

./bootstrap, src/doc/bootstrap: Auto-generate apt-get/yum command lines

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

ce34ce1

Makefile: Add dependencies on build/pkgs/*.txt build/pkgs/*/distros/*.txt

[mkoeppe]

comment:31

Replying to @mkoeppe:

Replying to @jhpalmieri:

How is this supposed to work with an incremental build?

Thanks for catching this -- I need to add some dependencies

Done.

[jhpalmieri]

comment:32

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

Is there a ticket which adds homebrew information to the installation/source.rst?

[mkoeppe]

comment:33

Replying to @jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The generated files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

[mkoeppe]

comment:34

Replying to @jhpalmieri:

Is there a ticket which adds homebrew information to the installation/source.rst?

#29104 is for homebrew. I haven't added the package information there yet -- any help is very welcome!

[jhpalmieri]

comment:35

Replying to @mkoeppe:

Replying to @jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation.

The generated files are in src/doc/en/installation. The source files are in build/pkgs/. Hence the dependency that I added. Changes to the .txt files trigger bootstrapping.

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

$ git checkout t/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual 
Switched to branch 't/26964/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual'
$ make configure
make: `configure' is up to date.

./bootstrap -d generates the correct files, but I have to run it manually.

[dimpase]

comment:36

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

can we get rid of configure target?

[dimpase]

comment:37

Replying to @jhpalmieri:

That doesn't help, because the necessary text files are not build/pkgs/*.txt build/pkgs/*/distros/*.txt but rather in src/doc/en/installation. By the way, I'm not sure whether I like the old style

     $ sudo apt-get install binutils pixz gcc g++ gfortran make m4 perl tar \
       git patch openssl libssl-dev libz-dev bc libbz2-dev liblzma-dev libgmp-dev \
       libffi-dev libgf2x-dev libcurl4-openssl-dev libzmq3-dev curl yasm \
       pkg-config libntl-dev libmpfr-dev libmpc-dev libflint-dev \
       libpcre3-dev libgd-dev libflint-dev libflint-arb-dev \
       libsymmetrica2-dev gmp-ecm libecm-dev libisl-dev libgivaro-dev \
       libpari-dev pari-gp2c libec-dev liblrcalc-dev \
       libm4ri-dev libm4rie-dev liblfunction-dev lcalc \
       libopenblas-dev r-base-dev libgsl-dev libcliquer-dev cliquer

or the new style

    $ sudo yum install binutils boost-devel bzip2 bzip2-devel curl eclib-devel findutils gcc gcc gcc-c++ gcc-c++ gcc-gfortran gcc-gfortran gfan git givaro-devel gmp gmp-devel gmp-ecm-devel libcurl-devel libmpc-devel lrcalc-devel m4 m4ri-devel m4rie-devel make mpfr-devel nauty ncurses-devel ntl-devel openblas-devel pari-devel pari-elldata pari-galdata pari-galdata pari-galpol pari-gp pari-seadata patch perl perl-ExtUtils-MakeMaker pkg-config python3 readline-devel rw-devel symmetrica-devel tar which xz xz-devel yasm zlib-devel

better.

the new style is built automatically, whereas the old one was manual.
One can surely hack further to add insertion of \ after ~80 chars, but it's certainlty minor...

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

688c68e

Makefile [configure]: Add dependency on bootstrap, src/doc/bootstrap

[sagetrac-git]

Changed commit from ce34ce1 to 688c68e

[mkoeppe]

comment:39

Replying to @jhpalmieri:

But this ticket doesn't change any of those .txt files. In my case I have Sage 9.1.beta3, then I switch to this branch and make fails, even with your latest branch. Is there no way to avoid this?

Thanks!
I've added more dependencies to the make configure target that should take care of this case.

[mkoeppe]

comment:40

Replying to @dimpase:

the idea is that one can make configure is not a good one. configure is "built" by autoconf, not by make.

I don't think there's anything wrong with these make targets -- also autotools have them (in maintainer mode).

[dimpase]

comment:41

a typical autotools project won't have a Makefile in unbuilt state, only Makefile.am

[mkoeppe]

comment:42

That is true, but there are Makefile targets that regenerate configure etc. from configure.ac.

[jhpalmieri]

comment:43

That works for me, thanks.

I don't in general ever use make configure explicitly, but I was trying to get the previous versions of this branch to work so I tried it. With the most recent branch, I can just check it out and run make.

[dimpase]

comment:44

is sage-print-system-package-command meant to runnable by the user, or it's a "service" script?

[mkoeppe]

comment:45

It's an internal script of sage the distribution. That's why it's in build/bin.

[dimpase]

Reviewer: John Palmieri, Dima Pasechnik

[dimpase]

comment:46

OK, the only nitpck is that on modern Fedora's it's dnf, not yum.

[mkoeppe]

comment:47

Thank you!

[vbraun]

Changed branch from u/mkoeppe/__bootstrap__format_build_pkgs_spkg_distros__information_to_produce_the_apt_get_yum_command_lines_shown_in_the_installation_manual to 688c68e

[vbraun]

Changed commit from 688c68e to none

[vbraun]

comment:49

This breaks incremental builds: #29233