Allow different versions of python modules depending on the python version used

[antonio-rojas]

See https://groups.google.com/forum/#!topic/sage-packaging/i9VosTO9-cM for details.

Until python2 is dropped, many packages which have dropped python 2 support upstream (ipython, sphinx, scipy, networkx, matplotlib...) are stuck to old, unmaintained versions in sage. This will pose a particular problem for distros once Sage defaults to python 3, since those distros will probably be carrying newer versions of these dependencies and sage will have to be heavily patched to work with them.

The aim of this ticket is to allow different versions for python2/python3 packages, so the python3 versions can be kept up to date in sage itself and the relevant patches can be upstreamed.

CC: @videlec @timokau @kiwifb @embray @jdemeyer @fchapoton @vbraun

Component: packages: standard

Branch/Commit: public/ticket/28190 @ ecb7133

Issue created by migration from https://trac.sagemath.org/ticket/28190

[jhpalmieri]

comment:4

Do you have ideas about how to implement this? Maybe a directory build/pkgs/python2-packages/ and if building with Python 2, first look there for the directory, and if that fails, look in build/pkgs? Or instead, within build/pkgs/sphinx/ (for example), have a python2 directory with py2 build information. In any case, I think that if there is a default setting, it should be Python 3.

[antonio-rojas]

comment:5

Perhaps it would be simpler to keep the python2 and python3 versions as separate pkgs? (build/pkgs/sphinx vs build/pkgs/python2-sphinx)

[videlec]

comment:6

Replying to @antonio-rojas:

Perhaps it would be simpler to keep the python2 and python3 versions as separate pkgs? (build/pkgs/sphinx vs build/pkgs/python2-sphinx)

+1

It will also be simpler to remove the python2-* repos when the life of Python2 terminates.

[kiwifb]

comment:7

That follows some model of binary distros, and I am OK with that. Implementation will have some kinks as it relies on the relationship between the folder name, the name of the package for dependencies and the name of the source tarball. I would say the latest is the hardest bit to implement - dependencies not so much, it is just tedious work.

[fchapoton]

comment:8

Who is going to take care of that ?

• I would suggest not to change the name of the packages, not to duplicate.

• Rather have (when required) inside build/pkg folders some new files checksums-legacy.ini and package-version-legacy.txt

• Change the script in the appropriate place (where ?) so that it chooses the version to install according to the python version.

Erik, Jeroen : your opinions on the problem, please ?

[fchapoton]

Branch: public/ticket/28190

[fchapoton]

comment:9

as a kind of provocation, here is a sketch of tentative..

---

New commits:

ecb7133

first tentative of allowing a legacy version of packages

[fchapoton]

Commit: ecb7133

[jdemeyer]

Let's start by analyzing the problem and possible solutions before discussing implementation details.

For a given package, let V2 be the latest version supporting Python 2 and let V3 be the latest version supporting Python 3. There are several possibilities (assuming obviously that V3 >= V2):

(A) V2 == V3: the package continues to support Python 2.

(B) V2 < V3 but the versions are compatible: the functionality that Sage uses works the same way with both versions.

(C) V2 and V3 are incompatible.

In case (A), nothing need to be done, so we don't need to discuss this further. In case (B), it doesn't really matter what we do: even if a distro uses a different version from Sage, there is no problem. So we really need to worry only about case (C). So first of all a serious question: are there packages where (C) currently applies? If not, we're solving a non-problem.

For (C), I see three possible solutions:

(i) continue using V2 in Sage and leave it up to the distros to deal with that.

(ii) use V2 when compiling with Python 2 and V3 when compiling with Python 3.

(iii) use V3 and drop support for Python 2 in Sage completely.

This ticket proposes (ii) but I'm not convinced that this is what we should do. It means that code in the Sage library needs to deal with supporting two incompatible versions of a package. That will complicate the code and increase chances of breakage.

If we can wait, maybe (iii) is actually a viable option: Python 2 is officially unsupported half a year from now. If Sage fully works under Python 3 by then, maybe we can drop Python 2 support at that time.

[antonio-rojas]

comment:11

Replying to @jdemeyer:

So first of all a serious question: are there packages where (C) currently applies? If not, we're solving a non-problem.

Yes (see the linked sage-packaging post)

If we can wait, maybe (iii) is actually a viable option: Python 2 is officially unsupported half a year from now. If Sage fully works under Python 3 by then, maybe we can drop Python 2 support at that time.

That would be a solution, but would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

[jdemeyer]

comment:12

Replying to @antonio-rojas:

would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

What would that deprecation mean in practice?

From the user's point of view, Sage uses either Python 2 or Python 3 (whatever we default to at installation time), so there is going to be a sudden break anyway sooner or later. Once you're defaulting to Python 3, I don't know if we should still put effort in supporting Python 2.

[videlec]

comment:13

As mentioned by Samuel on the mailing list, there is a distinct situation where we might want to upgrade packages to Python3 before the transition py2 -> py3: the jupyter notebook. It is not exactly covered by the ticket description.

More concretely, we could

• python3 install the modules: jupyter_core, jupyter_client, traitlets, decorator, enum34, pyzmq, dateutil
• remove python2 installation of jupyter_core and jupyter_client (and dead dependencies)
• run the jupyter notebook under python3

Question: would that really be useful?

[timokau]

comment:14

Replying to @jdemeyer:

Replying to @antonio-rojas:

would you really consider going from defaulting to python2 to completely dropping it without any deprecation period?

What would that deprecation mean in practice?

From the user's point of view, Sage uses either Python 2 or Python 3 (whatever we default to at installation time), so there is going to be a sudden break anyway sooner or later. Once you're defaulting to Python 3, I don't know if we should still put effort in supporting Python 2.

+1 on that point. If users don't want to migrate right away, they can continue using an older sage version. That may have security implications of course, but since some dependencies won't receive any updates anyway those implications are there either way.

There is little point to the "python2 deadline" if half of the projects decide "just a little longer".

[embray]

comment:15

Replying to @jdemeyer:

This ticket proposes (ii) but I'm not convinced that this is what we should do. It means that code in the Sage library needs to deal with supporting two incompatible versions of a package. That will complicate the code and increase chances of breakage.

I don't think that's such a problem. Yes, it's annoying. But with care, it can be done fairly cleanly, and explicitly supporting multiple versions of a depdency can be more stable.

E.g. in Astropy we support something like 4 different Numpy versions, all of which occasionally need different version-specific worksarounds or features. This is usually done explicitly in the code with pre-defined constants like if NUMPY_LT_1_11: ....

[embray]

comment:16

Ticket retargeted after milestone closed

[antonio-rojas]

comment:17

Now that 9.0 is out, what is the plan for this? Is python2 going to be dropped for 9.1? My current ipython 7 patch is almost 5000 LOC and getting harder to maintain, it would be nice to upstream it. Also, updating to python 3.8 would be more complicated if python2 support is to be kept, see #27754 comment:55

[embray]

comment:18

I don't think there should be a rush to upgrade to Python 3.8. With the first Python 3-based release just out, let's give it a little breathing room for things to shake out, as I'm sure it will still be a rocky transition for some users at first (hopefully not too much so though).

[mkoeppe]

comment:19

It seems we are dropping Python2 support after 9.1. See #29141 for the reference to the discussion on sage-devel.

I propose to close this ticket, in favor of #28000: Remove python 2 support

[fchapoton]

comment:20

I agree, let us move forward.