Normal Python packages: postpone wheel installation to the post-install phase

[mkoeppe]

Previously, on sdh_pip_install, the wheel file is staged in DESTDIR, but the wheel is installed immediately.
Now we store a new script spkg-pipinst, which is run after unloading DESTDIR (and before any spkg-postinst script), which does the actual installation of the wheel.

• This resolves sdh_store_and_pip_install_wheel: Install from the persistent wheel directory #30956 (fixing the wheel URLs shown in sage -pip freeze) -- except when SAGE_SUDO is set.

Apart from this and some changes to the messages displayed during package installation, this should make no difference for any of our packages.

Just so that it is tested for at least one package in CI, we include a small package update.

Together with

• build/bin/sage-spkg: Add options for checking, cleaning separately from build/install #36738 and
• Refactor bootstrap-conda, m4/sage_spkg_collect.m4 through sage-package; handle dependencies_build #36740,

this is preparation for requiring only the build dependencies ("build-system requires") while building a wheel for the package, and to require the runtime dependencies ("install-requires") only later, when the wheel is to be installed.

📝 Checklist

• The title is concise, informative, and self-explanatory.
• The description explains in detail what this PR is about.
• I have linked a relevant issue or discussion.
• I have created tests covering the changes.
• I have updated the documentation accordingly.

⌛ Dependencies

[jhpalmieri]

pip failed to build:

real	0m0.751s
user	0m0.370s
sys	0m0.253s
Copying package files from temporary location /Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/var/tmp/sage/build/pip-23.3.1/inst to /Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11
Running pip-install script for pip-23.3.1.
/Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
(ignoring error)
/Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
Warning: installing with "python3 -m pip install --verbose --no-index --find-links=/Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels --disable-pip-version-check --isolated --no-cache-dir" failed. Retrying, adding "--no-deps --ignore-installed --ignore-requires-python"
/Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
Error: installing with pip  failed
********************************************************************************
Error installing --ignore-installed -r
/Users/jpalmier/Sage/TESTING/sage-10.2.rc4/local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/pip/spkg-requirements.txt
********************************************************************************
************************************************************************
Error running the pipinst script for pip-23.3.1.
************************************************************************

[mkoeppe]

In hindsight it's obvious that I should have tested this case...

[jhpalmieri]

I don't typically set SAGE_SUDO, but to test I set it to 'sudo -E' and once again, pip failed to install:

removing /var/folders/7c/bz8vl5tx47ndyy8knwrtg1k00000gp/T/tmp.Y4ofb1qD0R
/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
(ignoring error)
/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
Warning: installing with "python3 -m pip install --verbose --no-index --find-links=/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels --disable-pip-version-check --isolated --no-cache-dir" failed. Retrying, adding "--no-deps --ignore-installed --ignore-requires-python"
/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/bin/python3: No module named pip
Error: installing with pip  failed
********************************************************************************
Error installing
--root=/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/tmp/sage/build/pip-23.3.1/inst
--no-warn-script-location --ignore-installed -r
/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/lib/sage/scripts/pip/spkg-requirements.txt

[github-actions]

Documentation preview for this PR (built with commit f981b15; changes) is ready! 🎉

[mkoeppe]

I don't typically set SAGE_SUDO, but to test I set it to 'sudo -E' and once again, pip failed to install

Thanks for testing! Fixed now.

[jhpalmieri]

Maybe things with SAGE_SUDO="sudo -E" still aren't right. With this branch but without setting SAGE_SUDO, just building normally, ./sage --pip freeze | grep tmp returns no hits, but with SAGE_SUDO="sudo -E", I get lots of hits:

% ./sage --pip freeze | grep tmp | wc
     140     420   50093

This looks like the same as building without this branch. Examples with this branch + setting SAGE_SUDO:

alabaster @ file:///Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/tmp/sage/build/alabaster-0.7.12/inst/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/alabaster-0.7.12-py2.py3-none-any.whl#sha256=97a89e510af23283d07834c7760e08086d4f8813597e860b6f855ff3c09e7863
appnope @ file:///Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/tmp/sage/build/appnope-0.1.3/inst/Users/jpalmier/Sage/TESTING/test/sage-10.2.rc5/local/var/lib/sage/venv-python3.11/var/lib/sage/wheels/appnope-0.1.3-py2.py3-none-any.whl#sha256=3a530d0a94cfac4ba3187c16266dbb3d338216726a374c94dcaeb999cfe68b37

[mkoeppe]

You are right. I do not have a solution for fixing wheel URL for the SAGE_SUDO case. I have updated the PR description.

[jhpalmieri]

Okay. As far as I can tell, this doesn't make anything worse (or even really change anything) if SAGE_SUDO is set, and it is an improvement if SAGE_SUDO is not set. Let's merge it.

[mkoeppe]

Thank you!

[mkoeppe]

At some point we may want to deprecate SAGE_SUDO. Introducing it was helpful as it informed restructuring the installation of packages into build vs. install phase. But my own use case for this feature (for use by IT on a shared installation) has gone away, and I'm not sure if this is used by others.