New option "./configure --enable-wheels" #32874

mkoeppe · 2021-11-15T02:28:03Z

When ./configure --disable-editable is in use (the default before #32406), the Sage library is installed using its custom incremental implementation of setup.py install -- which installs its files in site-packages and then removes the files that it does not know about (the "cleaner").

Problem 1: The cleaner is incompatible with namespace packages and trying to fix it would lead to complicated and awkward code (#32927). (Besides, setup.py install is deprecated since https://setuptools.pypa.io/en/latest/history.html#v58-3-0)

Problem 2: Because of the direct installation, we do not have wheels for sagemath-standard available (even if make wheels is used explicitly). Such wheels can be useful for making separate venvs.

In this ticket:

We create a new option ./configure --enable-wheels (which can be used both with ./configure --enable-editable and ./configure --disable-editable).
If enabled, installation of sagelib will go through wheel building and installation - like we do with all other Python packages. This enables modularized builds (Refactor sagemath-standard through sagemath-{categories,environment,repl} #34587).
If disabled, there is no change.

A trivial ./sage -b is much slower with ./configure --disable-editable --enable-wheels compared to ./configure --disable-editable (from 28s as per timings in #32406 comment:21 to 70s).
This ticket brings a number of speed improvements to mitigate this.

CC: @jhpalmieri

Component: build

Author: Matthias Koeppe

Branch: 25898ba

Reviewer: Kwankyu Lee

Issue created by migration from https://trac.sagemath.org/ticket/32874

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-11-15T02:45:07Z

Branch: u/mkoeppe/remove_use_of__setup_py_install__for_sagelib

mkoeppe · 2021-11-15T03:12:20Z

New commits:

`b9b1dba`	`Enable editable mode also for other sage packages`
`a257621`	`Add error handling`
`89bc3ee`	`Merge branch 'develop' of git://github.com/sagemath/sage into public/build/inplace_ext`
`7bd6ce4`	`Partly revert "Enable editable mode also for other sage packages"`
`3ef9c1d`	`Merge #32713`
`50dd26c`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`b5c40a8`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`

mkoeppe · 2021-11-15T03:12:20Z

Author: Matthias Koeppe

mkoeppe · 2021-11-15T03:12:20Z

Commit: b5c40a8

mkoeppe · 2022-05-12T02:10:33Z

Changed dependencies from #32713 to #32406

sagetrac-git · 2022-07-22T19:03:13Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`8619b29`	`configure.ac: Make --enable-editable the default`
`6513b6b`	`README.md: Explain configure --disable-editable`
`ff8710e`	`docker/Dockerfile: Use configure --disable-editable`
`7221a76`	`src/setup.py: Do not run find_namespace_packages for 'setup.py dist_info'`
`a911e0f`	`src/MANIFEST.in: prune sage_docbuild, doc`
`edfa3b4`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`f7ea602`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`

sagetrac-git · 2022-07-22T19:03:13Z

Changed commit from b5c40a8 to f7ea602

mkoeppe · 2022-07-22T19:35:12Z

comment:10

[sagelib-9.7.beta5] sage-flock: error: argument LOCK: can't create '/doesnotexist/var/lock': [Errno 30] Read-only file system: '/doesnotexist'

sagetrac-git · 2022-07-22T21:03:13Z

Branch pushed to git repo; I updated commit sha1. New commits:

`df1e7eb`	`build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`f358962`	`build/pkgs/sagelib/spkg-install: No need to set env var PYTHON`
`c761788`	`build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`99d4129`	`build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation`

sagetrac-git · 2022-07-22T21:03:13Z

Changed commit from f7ea602 to 99d4129

sagetrac-git · 2022-09-26T05:01:26Z

Branch pushed to git repo; I updated commit sha1. This was a forced push. New commits:

`07f5cde`	`build/pkgs/sagelib/spkg-install: Use helper function sdh_pip_editable_install`
`3cdf86e`	`build/pkgs/sagelib/spkg-install: In non-editable mode, Use sdh_pip_install instead of 'setup.py install'`
`459ee2f`	`build/bin/sage-pip-install: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`3de2608`	`build/pkgs/sagelib/spkg-install: No need to set env var PYTHON`
`e6eae68`	`build/bin/sage-pip-uninstall: Put lock file in SAGE_VENV, not SAGE_LOCAL`
`e829b08`	`build/pkgs/sagelib/spkg-install [SAGE_EDITABLE=no]: Use --no-build-isolation`

sagetrac-git · 2022-09-26T05:01:26Z

Changed commit from 99d4129 to e829b08

kwankyu · 2022-10-03T11:45:13Z

comment:64

Replying to Kwankyu Lee:

./configure --enable-wheels doesn't seem to work for me, even after ./bootstrap (is this necessary?)

sage -b after ./configure --disable-editable does not install wheel. sage -b just takes 20 seconds.

mkoeppe · 2022-10-03T16:24:53Z

comment:65

Yes, there are 4 modes of operation now, all combinations of --{en,dis}able-editable and --{en,dis}able-wheels should work

kwankyu · 2022-10-04T04:00:11Z

comment:66

What doesn't work is to configure those options separately: ./configure --enable-wheels and then ./configure --disable-editable. I guess this is normal.

Yes, ./configure --disable-editable --enable-wheels does build the wheel :)

mkoeppe · 2022-10-04T04:03:06Z

comment:67

To add options to the previous list of configure options, there's unfortunately no shorter idiom than

eval ./configure $(./config.status --config) NEW-OPTIONS...

kwankyu · 2022-10-04T04:13:36Z

comment:68

Making --enable-wheels default eventually is necessary for sagelib modularization. Right?

mkoeppe · 2022-10-04T04:21:38Z

comment:69

I'd say it will be necessary as soon as the first external packages arrive that declare a dependency on a modularized distribution rather than on sagemath-standard.

Until then we can get away with keeping both --enable-editable and --disable-editable --disable-wheels monolithic; and we can use --disable-editable --enable-wheels for working on the details of the modularized distributions.

mkoeppe · 2022-10-04T04:33:04Z

comment:70

(see ticket description of #34587)

kwankyu · 2022-10-04T05:45:36Z

comment:71

Questions on changes in src/MANIFEST.in:

Is this line include MANIFEST.in necessary? Perhaps because src/MANIFEST.in is a symlink?
How was this list made ? Could the hard-coded list be still valid in future?

include sage/modular/arithgroup/farey_symbol.h
include sage/cpython/debugimpl.c
include sage/graphs/base/boost_interface.cpp
include sage/graphs/cliquer/cl.c
include sage/graphs/graph_decompositions/sage_tdlib.cpp
include sage/libs/eclib/wrap.cpp
include sage/libs/linkages/padics/relaxed/flint_helper.c
include sage/misc/inherit_comparison_impl.c
include sage/modular/arithgroup/farey.cpp
include sage/modular/arithgroup/sl2z.cpp
include sage/rings/bernmm/bern_modp.cpp
include sage/rings/bernmm/bern_modp_util.cpp
include sage/rings/bernmm/bern_rat.cpp
include sage/rings/bernmm/bernmm-test.cpp
include sage/rings/padics/transcendantal.c
include sage/rings/polynomial/weil/power_sums.c
include sage/schemes/hyperelliptic_curves/hypellfrob/hypellfrob.cpp
include sage/schemes/hyperelliptic_curves/hypellfrob/recurrences_ntl.cpp
include sage/schemes/hyperelliptic_curves/hypellfrob/recurrences_zn_poly.cpp
include sage/stats/distributions/dgs_bern.c
include sage/stats/distributions/dgs_gauss_dp.c
include sage/stats/distributions/dgs_gauss_mp.c
include sage/symbolic/ginac/*.cpp

kwankyu · 2022-10-04T05:45:46Z

Reviewer: Kwankyu Lee

sagetrac-git · 2022-10-04T18:15:49Z

Changed commit from 11507a5 to 974c7fd

sagetrac-git · 2022-10-04T18:15:49Z

Branch pushed to git repo; I updated commit sha1. New commits:

`974c7fd`	`src/MANIFEST.in: Remove redundant entries for MANIFEST.in, pyproject.toml`

mkoeppe · 2022-10-04T18:16:37Z

comment:74

Replying to Kwankyu Lee:

Questions on changes in src/MANIFEST.in:

Is this line include MANIFEST.in necessary? Perhaps because src/MANIFEST.in is a symlink?

No, not necessary any more; it probably was necessary with prehistoric versions of setuptools.

sagetrac-git · 2022-10-04T18:26:27Z

Branch pushed to git repo; I updated commit sha1. New commits:

`25898ba`	`src/MANIFEST.in: Exclude generated file farey_symbol.h; add comments`

sagetrac-git · 2022-10-04T18:26:27Z

Changed commit from 974c7fd to 25898ba

mkoeppe · 2022-10-04T18:27:11Z

comment:76

Replying to Kwankyu Lee:

How was this list made ?

I've added a comment (and fixed a mistake that I had made along the way).

kwankyu · 2022-10-04T23:27:31Z

comment:77

Thanks.

It works well and looks good to me.

mkoeppe · 2022-10-04T23:58:30Z

comment:78

Thank you!

vbraun · 2022-10-16T22:16:00Z

Changed branch from u/mkoeppe/remove_use_of__setup_py_install__for_sagelib to 25898ba

antonio-rojas · 2022-10-17T20:14:15Z

Changed commit from 25898ba to none

antonio-rojas · 2022-10-17T20:14:15Z

comment:80

This makes tests depend on sage_setup, so one gets failures when testing an installed sage

sage -t --long --random-seed=334405888414880658562662015982432628863 /usr/lib/python3.10/site-packages/sage/misc/package_dir.py
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 108, in sage.misc.package_dir.read_distribution
Failed example:
    from sage_setup.find import read_distribution
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[1]>", line 1, in <module>
        from sage_setup.find import read_distribution
    ModuleNotFoundError: No module named 'sage_setup'
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 109, in sage.misc.package_dir.read_distribution
Failed example:
    read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[2]>", line 1, in <module>
        read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'tdlib.pyx'))
    NameError: name 'read_distribution' is not defined
**********************************************************************
File "/usr/lib/python3.10/site-packages/sage/misc/package_dir.py", line 111, in sage.misc.package_dir.read_distribution
Failed example:
    read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
Exception raised:
    Traceback (most recent call last):
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 695, in _run
        self.compile_and_execute(example, compiler, test.globs)
      File "/usr/lib/python3.10/site-packages/sage/doctest/forker.py", line 1093, in compile_and_execute
        exec(compiled, globs)
      File "<doctest sage.misc.package_dir.read_distribution[3]>", line 1, in <module>
        read_distribution(os.path.join(SAGE_SRC, 'sage', 'graphs', 'graph_decompositions', 'modular_decomposition.py'))
    NameError: name 'read_distribution' is not defined
**********************************************************************

mkoeppe · 2022-10-17T20:54:33Z

comment:81

I've opened #34674 for this

kwankyu · 2022-10-17T22:17:53Z

comment:82

A question to educate myself: In what environment, sage_setup module is not available? Is this a linux package?

mkoeppe · 2022-10-17T22:21:06Z

comment:83

In distribution packaging (Antonio does the packaging for Arch Linux), sage_setup is typically not available at runtime.

kwankyu · 2022-10-17T22:27:19Z

comment:84

Okay. Thanks.

mkoeppe added this to the sage-9.5 milestone Nov 15, 2021

mkoeppe added c: build labels Nov 15, 2021

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.5, sage-9.6 Nov 15, 2021

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.6, sage-9.7 Mar 5, 2022

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Remove use of "setup.py install" for sagelib~~ Remove use of "setup.py install" for sagelib when ./configure --disable-editable is in use Jul 22, 2022

mkoeppe added s: needs review and removed s: needs review labels Jul 22, 2022

This comment has been minimized.

Sign in to view

mkoeppe modified the milestones: sage-9.7, sage-9.8 Jul 23, 2022

kwankyu added s: positive review and removed s: needs review labels Oct 4, 2022

vbraun removed the s: positive review label Oct 16, 2022

vbraun closed this as completed in a9abd3b Oct 16, 2022

isuruf mentioned this issue Jan 7, 2023

New modularized distributions sagemath-{sirocco,bliss,meataxe,mcqd,coxeter3,tdlib} #34346

Closed

vbraun mentioned this issue Oct 11, 2022

fix more W605 in rings #34645

Closed

New option "./configure --enable-wheels" #32874

New option "./configure --enable-wheels" #32874

Comments

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

mkoeppe commented Nov 15, 2021

This comment has been minimized.

mkoeppe commented May 12, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

mkoeppe commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

sagetrac-git mannequin commented Jul 22, 2022

This comment has been minimized.

sagetrac-git mannequin commented Sep 26, 2022

sagetrac-git mannequin commented Sep 26, 2022

kwankyu commented Oct 3, 2022

mkoeppe commented Oct 3, 2022

kwankyu commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

kwankyu commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

kwankyu commented Oct 4, 2022

kwankyu commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

sagetrac-git mannequin commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

kwankyu commented Oct 4, 2022

mkoeppe commented Oct 4, 2022

vbraun commented Oct 16, 2022

antonio-rojas commented Oct 17, 2022

antonio-rojas commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022

mkoeppe commented Oct 17, 2022

kwankyu commented Oct 17, 2022