relocatable wheel version of package sage_conf #31396

mkoeppe · 2021-02-14T22:06:47Z

This version of sage_conf is for making a wheel that packages the precompiled non-Python bits of the Sage distribution, making SAGE_ROOT (and thus SAGE_LOCAL=$SAGE_ROOT/local) relocatable using Marc Culler's symbolic link surgery (the method proposed in #31076, using SAGE_ROOT=/var/tmp/sage-... and a symlink).

The sage script invokes sage-config to determine SAGE_ROOT and SAGE_LOCAL. In the version of sage-config supplied by this version of sage_conf, we ensure that the symlink from /var/tmp/sage-.... to the actual install location is set.

So far, a wheel has been built for XCode python 3.8 on macOS 10.15. The wheel is 670MB in size, an order of magnitude above the standard file size limit on PyPI; but a file size limit increase (requested in pypi/support#985) has kindly been granted by the PyPI team.

The wheel declares dependencies (install-requires) to all Python packages in the Sage distribution that have extension modules. The dependencies are specific using @ to URLs on
https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2, where I have uploaded the binary wheels. Unfortunately, by PyPI policy, such @ references are not allowed for packages on PyPI.
Hence, the pip invocation needs to use a URL.

(See also https://twitter.com/mkoeppe_math/status/1378860285537054723)

Instructions for testing:

Make sure that the bin directory of the user installation scheme is in PATH. For example, on macOS:

$ export PATH=$HOME/Library/Python/3.8/bin:/usr/bin:/bin:/usr/sbin:/sbin

Install the wheel in the user scheme:

$ /usr/bin/python3 -m pip install -U --user  https://github.com/sagemath/sage-wheels/releases/download/9.3.rc1/sagemath_standard-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl

Run the installed sage.

$ sage

Procedure for building the sage_conf wheel and compatible wheels:

For building wheels on macOS Catalina that uses XCode python 3.8:

$ SKIP_CONFIGURE=1 tox -e local-macos-10.15-nohomebrew-python3_xcode -- bash
(tox -e ...) $ (cd src/pkgs/sage_conf-relocatable/ && python3 setup.py bdist_wheel --plat-name macosx_10_15_x86_64)

or

tox -e local-macos-10.15-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

This creates (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

src/pkgs/sage_conf-relocatable/dist/sage_conf-9.3rc1-cp38-cp38-macosx_10_15_x86_64.whl
src/pkgs/sage_conf-relocatable/sage_root/venv-cpython-38-darwin-macosx_10_15_x86_64/var/lib/sage/wheels/*.whl

Build for macOS Big Sur (requires a machine running Big Sur or Monterey) (uploaded to https://github.com/sagemath/sage-wheels/releases/tag/9.5.rc2)

tox -e local-macos-11.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for macOS Monterey (requires a machine running Monterey)

tox -e local-macos-12.1-nohomebrew-python3_xcode -- SAGE_CHECK=no sage-wheels

Build for Linux (does not work yet):

$ TARGETS_PRE=Makefile tox -e docker-manylinux-2_24-minimal-python3.9 -- sage-wheels

Follow-up steps:

also build a sagemath-standard wheel
build wheels for manylinux using tox -e docker-manylinux....
- wheel is much too big: 2.3GB
- auditwheel complains: RuntimeError: Invalid binary wheel, found the following shared library/libraries in purelib folder:; so we should replace our current abuse of package data (install_data) by platlib (install_lib). pypa/build on Linux - extension / binary modules in purelib pypa/packaging-problems#542 (comment): subclass build_ext instead of build_py
disable building the dependencies of the sphinx packages; remove unneeded static libraries; remove duplicated/triplicated shared libraries if possible -- for example by storing a tar file sage_root.tar (with symlinks) instead of including the sage root as a directory - https://docs.python.org/3/library/tarfile.html#tarfile.open
modularized relocatable wheel version of sage_conf #31602: split out sage_root from sage_conf and use install-requires in the sage_conf sdist/wheel with @ links to GH releases to the sage_root wheel and the built non-any wheels.

Depends on #33817

CC: @culler @kliem @dimpase @jhpalmieri @slel @kiwifb

Component: relocation

Author: Matthias Koeppe

Branch/Commit: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf @ 1b2d4be

Reviewer: Julian Rüth

Issue created by migration from https://trac.sagemath.org/ticket/31396

The text was updated successfully, but these errors were encountered:

mkoeppe · 2021-02-15T03:32:14Z

Branch: u/mkoeppe/relocatable_wheel_version_of_package_sage_conf

sagetrac-git · 2021-02-15T06:36:08Z

Commit: 9573b7c

sagetrac-git · 2021-02-15T06:36:08Z

Branch pushed to git repo; I updated commit sha1. New commits:

`952fb9a`	`Fix up package_data`
`9573b7c`	`Closer to working`

mkoeppe · 2021-02-15T06:40:47Z

Author: Matthias Koeppe

mkoeppe · 2021-02-15T07:56:12Z

comment:4

On macOS, some shared libraries from homebrew still leak into the build:

find src/pkgs/sage_conf-relocatable/.tox/python/lib/python3.9/site-packages/sage_root -name "*.so" -o -name "*.dylib" | xargs otool -L
libgiac.dylib:
/usr/local/opt/gettext/lib/libintl.8.dylib
R/library/cluster/libs/*.dylib:
	/usr/local/opt/gettext/lib/libintl.8.dylib (compatibility version 11.0.0, current version 11.0.0)
	/usr/local/opt/pcre/lib/libpcre.1.dylib (compatibility version 4.0.0, current version 4.12.0)
	/usr/local/opt/xz/lib/liblzma.5.dylib (compatibility version 8.0.0, current version 8.5.0)

Definitely should disable the R build (#30383).

Also check whether setting CC=gcc -sysroot.... helps getting rid of /usr/local leaking in.

Also need to add --enable-fat-binary.

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.
Also copytree from sage_root_source should be changed so it overwrites (updates) the existing contents of SAGE_ROOT.

the tox test should include the test with otool for self-containedness.

sagetrac-git · 2021-02-16T02:56:32Z

Changed commit from 9573b7c to f785d3c

sagetrac-git · 2021-02-16T02:56:32Z

Branch pushed to git repo; I updated commit sha1. New commits:

`f785d3c`	`src/pkgs/sage_conf-relocatable/tox.ini: Add environment macos-10.15`

mkoeppe · 2021-02-16T02:58:06Z

comment:6

Using -isysroot helps but giac configuration needs more work - #31403

mkoeppe · 2021-02-16T03:26:34Z

comment:7

Replying to @mkoeppe:

setup.py build_py should probably use a symlink from /var/tmp... into ~/.sage as well so that it is easy to test relocation after removing the symlink.

Does not work because sage resolves symlinks when it determines SAGE_ROOT.

mkoeppe · 2021-02-19T00:01:30Z

Changed dependencies from #29039 to #29039, #31409

sagetrac-git · 2021-02-19T00:02:07Z

Branch pushed to git repo; I updated commit sha1. New commits:

`6da7ba2`	`Add variable TARGETS`
`b2d4ab8`	`build/pkgs/{r,rpy2}: Downgrade to optional`
`8732076`	`src/sage/repl/ipython_tests.py: Mark R interface tests # optional - rpy2`
`dbdf084`	`src/sage/structure/sage_object.pyx: Mark R interface test # optional - rpy2`
`737b21c`	`src/sage/stats/r.py: Mark all 2 doctests in this file as # optional - rpy2`
`4403924`	`src/sage/interfaces/r.py: Mark all tests # optional - rpy2`
`e56a2a9`	`Merge branch 't/31409/cygwin_standard__r_build_fails_____downgrade_r__rpy2_to_optional' into t/31396/relocatable_wheel_version_of_package_sage_conf`

sagetrac-git · 2021-02-19T00:02:07Z

Changed commit from f785d3c to e56a2a9

sagetrac-git · 2021-02-19T02:19:30Z

Branch pushed to git repo; I updated commit sha1. New commits:

`a228974`	`src/pkgs/sage_conf-relocatable/setup.py: Move the SAGE_ROOT directory away after building`

sagetrac-git · 2021-02-19T02:19:30Z

Changed commit from e56a2a9 to a228974

sagetrac-git · 2021-02-19T03:35:35Z

Changed commit from a228974 to c260942

sagetrac-git · 2021-02-19T03:35:35Z

Branch pushed to git repo; I updated commit sha1. New commits:

`0cac965`	`build/pkgs/ecl: Update to 21.2.1, remove patches included in new version`
`6a62484`	`delete all merged upstream patches: MR 210, MR 231`
`62fac00`	`Merge branch 't/31336/upgrade_ecl_to_21_2_1' into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c260942`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-download-from-upstream-url`

sagetrac-git · 2021-02-19T06:10:23Z

Changed commit from c260942 to 4f17a04

sagetrac-git · 2021-02-19T06:10:23Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`dc5e225`	`build/pkgs/python3/spkg-configure.m4: On macOS, if the distutils test fails, try using empty ARCHFLAGS`
`df17ad4`	`tox.ini: Add configuration factors for specific homebrew python3.x`
`914ea58`	`build/pkgs/python3/spkg-configure.m4: Get rid of stray AC_SUBST`
`986739b`	`m4/sage_check_python_for_venv.m4: Fix reason`
`0b3e70d`	`build/pkgs/python3/spkg-configure.m4: If PYTHON_FOR_VENV is configured to build multiarch extensions, set SAGE_ARCHFLAGS to disable it`
`372adcb`	`.github/workflows/tox.yml: Update xcode versions`
`fc8b676`	`SAGE_CHECK_PYTHON_FOR_VENV: Rework with less nesting`
`5ae8044`	`Merge branch 'u/mkoeppe/accept__usr_bin_python3_from_xcode_12_3_on_macos_10_15__catalina_' of git://trac.sagemath.org/sage into t/31396/relocatable_wheel_version_of_package_sage_conf`
`c5c73a9`	`src/pkgs/sage_conf-relocatable/tox.ini: Define environments macos-python3.8, macos-python3.7, macos-python3_xcode`
`4f17a04`	`src/pkgs/sage_conf-relocatable/setup.py: Use --enable-fat-binary`

sagetrac-git · 2021-02-19T06:40:05Z

Branch pushed to git repo; I updated commit sha1. New commits:

`8514352`	`src/pkgs/sage_conf-relocatable/tox.ini: Run delocate-listdeps`

sagetrac-git · 2021-02-19T06:40:05Z

Changed commit from 4f17a04 to 8514352

sagetrac-git · 2021-03-15T06:02:00Z

Changed commit from 8514352 to 1baf390

sagetrac-git · 2021-03-15T06:02:00Z

Branch pushed to git repo; I updated commit sha1. Last 10 new commits:

`c3e4093`	`Rename SAGE_OPTIONAL_CLEANED_PACKAGES to SAGE_OPTIONAL_UNINSTALLED_PACKAGES`
`4916415`	`Merge tag '9.3.beta9' into t/30383/new_package_type__optional_enabled_by_default`
`a35428e`	`Merge #30383`
`e098133`	`src/pkgs/sage_conf/sage_conf.py.in: Add SAGE_SPKG_WHEELS`
`5dbd712`	`src/pkgs/sage_conf-pypi/setup.py: Use 'configure --disable-notebook', run 'make build' so that wheels are built`
`839c534`	`src/pkgs/sage_conf-pypi/setup.py: Fix typo`
`100ff51`	`src/pkgs/sage_conf-pypi/MANIFEST.in: Graft sage_root/build/pkgs/sage_docbuild/src`
`afa2ffd`	`Add configure option --disable-sagelib`
`a4f75b8`	`src/pkgs/sage_conf-pypi: Use configure --disable-sagelib, update README`
`1baf390`	`Merge branch 't/29039/pip_installable_package_sage_bootstrap' into t/31396/relocatable_wheel_version_of_package_sage_conf`

sagetrac-git · 2022-07-27T01:53:39Z

Changed commit from e4d36b5 to 2eeb402

sagetrac-git · 2022-07-27T01:53:39Z

Branch pushed to git repo; I updated commit sha1. New commits:

`ecc4a90`	`Merge tag '9.7.beta6' into t/33817/gh_actions__add_test_of_the_modularized_distributions`
`e82312a`	`pkgs/sagemath-{environment,repl}/tox.ini: Update from sagemath-objects`
`8358976`	`src/bin/sage-update-version: Update install-requires.txt for the distribution packages in pkgs/`
`b79b5c6`	`build/pkgs/sagemath_objects/dependencies: Restore comment lost in merge`
`7bb6dee`	`build/pkgs/python_build: Rename from build/pkgs/build`
`52e8c41`	`build/bin/sage-build-env[-config]: Do not override SAGE_EDITABLE if it is set already`
`64b212c`	`pkgs/sagemath-*/tox.ini: Make sagepython-norequirements actually use sage python`
`2eeb402`	`Merge #33817`

saraedum · 2022-12-14T22:35:30Z

comment:189

Sorry for not having a look at this ticket for a while. There are merge conflicts now btw.

sagetrac-git · 2022-12-15T01:16:24Z

Changed commit from 2eeb402 to 1b2d4be

sagetrac-git · 2022-12-15T01:16:24Z

Branch pushed to git repo; I updated commit sha1. New commits:

`1b2d4be`	`Merge tag '9.8.beta5' into t/31396/relocatable_wheel_version_of_package_sage_conf`

mkoeppe · 2023-09-29T22:08:35Z

Replaced by #36366.

mkoeppe added this to the sage-9.3 milestone Feb 14, 2021

mkoeppe added c: relocation labels Feb 14, 2021

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs info labels Jul 27, 2022

mkoeppe modified the milestones: sage-9.7, sage-9.8 Aug 31, 2022

saraedum added s: needs work and removed s: needs review labels Dec 14, 2022

mkoeppe modified the milestones: sage-9.8, sage-9.9 Jan 7, 2023

mkoeppe removed this from the sage-10.0 milestone Apr 30, 2023

mkoeppe added a commit to mkoeppe/sage that referenced this issue Sep 29, 2023

pkgs/sage-conf_relocatable: New (squashed from sagemath#31396)

4d98583

mkoeppe mentioned this issue Sep 29, 2023

pkgs/sage-conf_relocatable #36366

Draft

5 tasks

mkoeppe closed this as not planned Won't fix, can't repro, duplicate, stale Sep 29, 2023

github-actions bot removed the s: needs work label Sep 29, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

relocatable wheel version of package sage_conf #31396

relocatable wheel version of package sage_conf #31396

mkoeppe commented Feb 14, 2021

mkoeppe commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

This comment has been minimized.

mkoeppe commented Feb 15, 2021

mkoeppe commented Feb 15, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 16, 2021

sagetrac-git mannequin commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Mar 15, 2021

sagetrac-git mannequin commented Mar 15, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jul 27, 2022

sagetrac-git mannequin commented Jul 27, 2022

saraedum commented Dec 14, 2022

sagetrac-git mannequin commented Dec 15, 2022

sagetrac-git mannequin commented Dec 15, 2022

mkoeppe commented Sep 29, 2023

relocatable wheel version of package sage_conf #31396

relocatable wheel version of package sage_conf #31396

Comments

mkoeppe commented Feb 14, 2021

mkoeppe commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

sagetrac-git mannequin commented Feb 15, 2021

This comment has been minimized.

mkoeppe commented Feb 15, 2021

mkoeppe commented Feb 15, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 16, 2021

sagetrac-git mannequin commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 16, 2021

mkoeppe commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Feb 19, 2021

sagetrac-git mannequin commented Feb 19, 2021

This comment has been minimized.

sagetrac-git mannequin commented Mar 15, 2021

sagetrac-git mannequin commented Mar 15, 2021

This comment has been minimized.

sagetrac-git mannequin commented Jul 27, 2022

sagetrac-git mannequin commented Jul 27, 2022

saraedum commented Dec 14, 2022

sagetrac-git mannequin commented Dec 15, 2022

sagetrac-git mannequin commented Dec 15, 2022

mkoeppe commented Sep 29, 2023