spkg-configure.m4 for python3

[dimpase]

Do not install python3 if a sufficiently new version is available in $PATH. This could be a system python3, or the python3 from within a venv.

As suggested in #29032, we make $SAGE_LOCAL a venv (https://docs.python.org/3/library/venv.html) over the system python3 -- if a suitable system python3 is found -- by using the venv.EnvBuilder API.

This is done by the new script build/bin/sage-venv, which we use both during configure (to make sure that we select a system python3 for which venvs work correctly) and during make.

The venv does not include the system site-packages: We continue to install all Python packages into our venv using the existing build infrastructure. We keep the task of using system site-packages for the follow-up ticket #29023.

(Note, we use venv (new since Python 3.3), not virtualenv. So this change is limited to Python 3 builds of Sage.)

CC: @jdemeyer @embray @vbraun @kiwifb @jhpalmieri @videlec

Component: build: configure

Author: Matthias Koeppe, Erik Bray

Branch: 1c845a4

Reviewer: Dima Pasechnik

Issue created by migration from https://trac.sagemath.org/ticket/27824

[embray]

comment:3

As the Sage-8.8 release milestone is pending, we should delete the sage-8.8 milestone for tickets that are not actively being worked on or that still require significant work to move forward. If you feel that this ticket should be included in the next Sage release at the soonest please set its milestone to the next release milestone (sage-8.9).

[embray]

comment:4

For Python packages when installing using the system Python we definitely need a well thought out approach, likely with different options for handling questions such as where to install additional Python packages.

$SAGE_LOCAL already acts as a sort-of virtualenv, but we might still consider using virtualenv/venv in $SAGE_LOCAL (with the --system-site-packages option) as a way of extending sys.path while still using as many system Python packages as possible.

[mkoeppe]

comment:5

See #29013 - "In a python3 build, install all Python packages into a venv"

[mkoeppe]

Dependencies: #29013

[mkoeppe]

comment:6

Narrowed the ticket to py3.

[mkoeppe]

Branch: public/27824-python3-spkg-configure

[mkoeppe]

comment:8

Here's a version that works. The venv is created in build/make/Makefile target (in #29013, I created the venv in the spkg-postinst script of the python3 spkg.)

---

New commits:

1363425	In a python3 build, install all Python packages into a venv
a394268	sage-pip-install: Use PYTHON=sage-python23
a84815b	spkg-configure.m4 for python3

[mkoeppe]

Commit: a84815b

[mkoeppe]

Author: Matthias Koeppe

[dimpase]

comment:10

how about checking for Py3 deps: sqlite libpng bzip2 xz libffi ?
(I skipped in this list ones that are 2nd order, so no need to check them)

[dimpase]

comment:11

On Gentoo Python 3 does not have sqlite module by default. I had to do
$ pip install pysqlite3 --user, and still elliptic_curves spkg installation errors out.

[sagetrac-git]

Changed commit from a84815b to 0d0495f

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

0d0495f

build/pkgs/python3/spkg-configure.m4: Add SAGE_SPKG_DEPCHECK for sqlite libpng bzip2 xz libffi

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

98dff9d

Remove use of --system-site-packages

[sagetrac-git]

Changed commit from 0d0495f to 98dff9d

[mkoeppe]

comment:14

I've reduced the scope of the ticket, excluding the use of system site packages.

[sagetrac-git]

Changed commit from 98dff9d to 6180132

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

6180132

build/pkgs/python3/spkg-configure.m4: Find @PYTHON_FOR_VENV@ that has sqlite3 module

[dimpase]

comment:16

I tried this on gentoo (after a usual struggle to get sqlite3 module built in Python)
and it broke at docbuild, with

RuntimeError: libSingular not found--a working Singular install in $SAGE_LOCAL is required for Sage to work

even though libSingular has built just fine (it's some magic with from sage.env import SINGULAR_SO that is broken, so SINGULAR_SO is None)

[sagetrac-git]

Branch pushed to git repo; I updated commit sha1. New commits:

1c845a4

build/pkgs/python3/spkg-configure.m4: Do not use venv for python3 if we do a python2 build

[mkoeppe]

comment:99

Tests at https://github.com/mkoeppe/sage/actions/runs/61077298

[dimpase]

comment:100

Let's move this, looks good to me.

[dimpase]

Reviewer: Dima Pasechnik

[mkoeppe]

comment:101

Thanks!

[vbraun]

Changed branch from public/27824-python3-spkg-configure to 1c845a4

[mezzarobba]

Changed commit from 1c845a4 to none

[mezzarobba]

comment:103

Hello everyone,

When trying to upgrade to 9.1.rc5, I'm seeing an issue that I suspect could be a side effect of this ticket.

For some reason my sage/local/bin/python3.7 was a symlink to the system python binary. Sage decided it wanted to install its own python-3.7.3.p1, but that failed with a permission error on the symlink. After deleting the symlink, python builds and installs correctly. (I let the logs be overwritten, so I won't be able to provide much details — sorry.)

The overall Sage build still fails with an error about six not being found while building setuptools that I haven't yet investigated.

[dimpase]

comment:104

if you switch to py3 from py2, you probably need to run ./configure --with-python=3

it is not clear what version you were on, but the changes to the build system were very substantial, so ymmv.

[mkoeppe]

comment:105

Yes, this is a known limitation. I have created #29708 (Fix switching between --with-system-python3 and --without-system-python3) for it. We shall fix it in 9.2.