Update numpy to 1.23.5, scipy 1.9.3, networkx 2.8.8, meson_python 0.11.0 #34658

`d36ed98`	`tox.ini (python3.*-minimal): Provide system liblzma so that system python can be accepted; not system xz`
`f62b625`	`Merge #32182`
`8990d53`	`.github/workflows/docker.yml: Add option free_disk_space`
`09244b0`	`tox.ini (python3.x): Do not clobber user-provided EXTRA_SAGE_PACKAGES`
`9fb4eac`	`.github/workflows/docker.yml: Add option extra_sage_packages`
`3897adb`	`tox.ini (manylinux-minimal): Provide system liblzma so that system python can be accepted; not system xz`
`4b7f679`	`Merge #34081`
`fcc7087`	`build/pkgs/scipy: Update to 1.9.3`
`a2b5da4`	`build/pkgs/networkx: Update to 2.8.7`
`ba5c196`	`build/pkgs/pythran: Update to 0.12.0`

dimpase · 2022-10-31T09:09:54Z

comment:7

I get test errors in matrix2.pyx, line 6935 which calls scipy on these tasks:

A = matrix(RDF, [[1, -2], [3, 4]])
B = matrix(RDF, [[0, 7], [2, -3]])
D, P = A.eigenmatrix_left(B)
(P * A - D * P * B).norm() < 1e-14

(and a similar at line 7151 (for _right rather than _left).

And it's not just numerical noise:

sage: (P * A - D * P * B).norm()
12.951156314021555
sage: P * A - D * P * B
[ 1.1102230246251565e-16     -12.951156314021551]
[-2.7755575615628914e-17 -3.1086244689504383e-15]

dimpase · 2022-10-31T09:47:24Z

comment:8

the 1st of two gen. eigenvectors is wrong.

sage: A.eigenvectors_left(B)
[(-2.1075759441619084, [(-0.9905315991237724, 0.1372849268394109)], 1),
 (-0.3922809560592623, [(-0.9668186179746967, 0.2554638133620835)], 1)]

vs. the previous version (1.8.1), correct one:

sage: A.eigenvectors_left(B)
[(1.8208523846306908, [(-0.5400711566794881, -0.8416193591659354)], 1),
 (-0.39228095605926194, [(-0.9668186179746967, 0.2554638133620835)], 1)]

This can be traced down to scipy call:

sage:         v, eig = scipy.linalg.eig(A.numpy(),
....:                                   B.numpy(),
....:                                   right=False, left=True,
....:                                   homogeneous_eigvals=False)
sage: v
array([-2.10757594+0.j, -0.39228096+0.j])
sage: eig
array([[-0.9905316 , -1.        ],
       [ 0.13728493,  0.26423138]])

vs, correct, previous version:

sage: v
array([ 1.82085238+0.j, -0.39228096+0.j])
sage: eig
array([[-0.54007116, -1.        ],
       [-0.84161936,  0.26423138]])

dimpase · 2022-10-31T10:32:03Z

Upstream: Not yet reported upstream; Will do shortly.

dimpase · 2022-10-31T10:32:03Z

comment:9

I can reproduce this with "plain" scipy on Gentoo (installed as system Gentoo package)

dimpase · 2022-10-31T10:46:23Z

comment:10

This is !Gentoo/Sage-specific. If instead of system- (or Sage-)built scipy I use binary wheels from pypi, everything is fine.

The next test will be to try building Sage's scipy on a non-Gentoo system.

dimpase · 2022-10-31T11:57:16Z

comment:11

cc-ing Gentoo people. On Gentoo-built scipy 1.9.3, one gets incorrect generalised eigenvectors.

$ python3
Python 3.10.7 (main, Oct 10 2022, 13:00:40) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import scipy, numpy
>>> A=numpy.array([[ 1., -2.],[ 3.,  4.]])
>>> B=numpy.array([[ 0.,  7.],[ 2., -3.]])
>>> scipy.linalg.eig(A,B,right=False, left=True, homogeneous_eigvals=False)
(array([-2.10757594+0.j, -0.39228096+0.j]), array([[-0.9905316 , -1.        ],
       [ 0.13728493,  0.26423138]]))

instead of correct

(array([ 1.82085238+0.j, -0.39228096+0.j]), array([[-0.54007116, -1.        ],
       [-0.84161936,  0.26423138]]))

pip-installing binary wheels for numpy ans scipy from PyPI (same versions) produces correct results.

dimpase · 2022-10-31T12:01:16Z

comment:12

On Fedora 34, everything is fine (with openblas 0.3.19, not sure whether it matters)

dimpase · 2022-10-31T13:53:23Z

comment:13

Building Sage's OpenBLAS (Gentoo provides an older OpenBLAS, 0.3.20) appears to fix the problem.
Should we ban 0.3.20 ? Anyhow, Gentoo has to fix it.

dimpase · 2022-10-31T13:57:13Z

comment:14

Submitted https://bugs.gentoo.org/878883

dimpase · 2022-10-31T13:57:26Z

Changed upstream from Not yet reported upstream; Will do shortly. to Reported upstream. No feedback yet.

dimpase · 2022-10-31T14:05:53Z

Changed upstream from Reported upstream. No feedback yet. to Reported upstream. Developers acknowledge bug.

dimpase · 2022-10-31T18:05:08Z

comment:17

Further on arm64 (Apple M1) one gets

...
[scipy-1.9.3]   [1224/1224] scipy/io/_harwell_boeing/tests/test_hb.py
[scipy-1.9.3]   Preparing metadata (pyproject.toml): finished with status 'done'
[scipy-1.9.3] Building wheels for collected packages: scipy
[scipy-1.9.3]   Building wheel for scipy (pyproject.toml): started
[scipy-1.9.3]   Running command Building wheel for scipy (pyproject.toml)
[scipy-1.9.3]   Building wheel for scipy (pyproject.toml): finished with status 'done'
[scipy-1.9.3]   Created wheel for scipy: filename=scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl size=21560240 sha256=7f146f5c01cacccb1f543dd03f7852c9fa69832ac83e3438a3054b07ddadf986
[scipy-1.9.3]   Stored in directory: /private/var/folders/81/vt06lnj17rj0cpc9gdshy4q00000gr/T/pip-ephem-wheel-cache-u972yyp0/wheels/07/68/a7/3970e07638c02f51cea93e74047bd18f05b865d87665dddaa5
[scipy-1.9.3] Successfully built scipy
[scipy-1.9.3] WARNING: Skipping scipy as it is not installed.
[scipy-1.9.3] Using pip 22.2.2 from /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/pip (python 3.10)
[scipy-1.9.3] Looking in links: /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels
[scipy-1.9.3] ERROR: scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl is not a supported wheel on this platform.
[scipy-1.9.3] Warning: installing with "python3 -m pip install --verbose --no-index --find-links=/Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels --disable-pip-version-check --isolated --no-cache-dir" failed. Retrying, adding "--no-deps --ignore-installed --ignore-requires-python"
[scipy-1.9.3] Using pip 22.2.2 from /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/lib/python3.10/site-packages/pip (python 3.10)
[scipy-1.9.3] Looking in links: /Volumes/dima/sage/local/var/lib/sage/venv-python3.10/var/lib/sage/wheels
[scipy-1.9.3] ERROR: scipy-1.9.3-cp310-cp310-macosx_13_arm64.whl is not a supported wheel on this platform.
[scipy-1.9.3] Error: installing with pip  failed

dimpase · 2022-10-31T18:30:35Z

comment:18

upgrading to pip 22.3 does not help

mkoeppe · 2022-11-21T21:05:04Z

Changed upstream from Reported upstream. Developers acknowledge bug. to Fixed upstream, in a later stable release.

sagetrac-git · 2022-11-21T23:57:22Z

Branch pushed to git repo; I updated commit sha1. New commits:

`d74528c`	`Merge tag '9.8.beta4' into t/34658/update_numpy_to_1_23_4`

sagetrac-git · 2022-11-21T23:57:22Z

Changed commit from bbf5f95 to d74528c

jhpalmieri · 2022-11-22T21:58:48Z

comment:29

On OS X M2, scipy fails some tests in its test suite, but installation succeeds anyway, not sure why.

=========================== short test summary info ============================
FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: 
FAILED optimize/tests/test_linprog.py::TestLinprogIPSparse::test_bug_6139 - A...
FAILED optimize/tests/test_linprog.py::TestLinprogIPSparsePresolve::test_bug_6139
FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[gmres]
= 4 failed, 37207 passed, 3314 skipped, 12061 deselected, 136 xfailed, 12 xpassed, 326 warnings in 515.97s (0:08:35) =

jhpalmieri · 2022-11-22T21:59:55Z

comment:30

Test suite passes on OS X Intel.

mkoeppe · 2022-11-22T23:07:52Z

comment:31

One of them may be scipy/scipy#16926

jhpalmieri · 2022-11-22T23:10:09Z

comment:32

I would prefer that a failing test suite meant a failed installation, but maybe python3 -c 'import scipy; scipy.test()' doesn't return a nonzero code if it fails.

mkoeppe · 2022-11-22T23:11:27Z

comment:33

I agree, we should fix this.

jhpalmieri · 2022-11-22T23:18:01Z

comment:34

Maybe we should merge this now and patch things up in follow-up tickets, since I (for one) can't build the most recent beta without this. Are there other issues that should be addressed on this ticket?

jhpalmieri · 2022-11-22T23:18:45Z

comment:35

(I took a quick glance at the scipy code, and it looks like it's trying to return different exit codes. It was only a very quick glance, though.)

mkoeppe · 2022-11-22T23:19:56Z

comment:36

Tests for Linux are still running at https://github.com/mkoeppe/sage/actions/runs/3519527763

mkoeppe · 2022-11-23T04:27:24Z

comment:38

There is a problem on archlinux-standard related to scipy; that is unchanged by this upgrade. That's #34779

mkoeppe · 2022-11-23T07:34:28Z

comment:39

The other Linux platforms look fine.

jhpalmieri · 2022-11-23T18:10:56Z

Changed reviewer from https://github.com/mkoeppe/sage/actions/runs/3251926884 to John Palmieri

jhpalmieri · 2022-11-23T18:10:56Z

comment:40

Okay, then let's proceed here.

mkoeppe · 2022-11-23T19:42:27Z

comment:41

Thanks!

dimpase · 2022-12-08T14:10:49Z

comment:42

blocker, as on a recent macOS one cannot build scipy (1.9.20.

vbraun · 2022-12-11T11:11:52Z

Changed branch from u/mkoeppe/update_numpy_to_1_23_4 to d74528c

mkoeppe added this to the sage-9.8 milestone Oct 14, 2022

mkoeppe added c: packages: standard labels Oct 14, 2022

mkoeppe added the s: needs review label Oct 14, 2022

This comment has been minimized.

Sign in to view

mkoeppe changed the title ~~Update numpy to 1.23.4~~ Update numpy to 1.23.4, scipy 1.9.3, networkx 2.8.7 Oct 22, 2022

dimpase added s: needs work and removed s: needs review labels Oct 31, 2022

This comment has been minimized.

Sign in to view

mkoeppe added s: needs review and removed s: needs work labels Nov 21, 2022

mkoeppe changed the title ~~Update numpy to 1.23.4, scipy 1.9.3, networkx 2.8.8~~ Update numpy to 1.23.5, scipy 1.9.3, networkx 2.8.8, meson_python 0.11.0 Nov 21, 2022

mkoeppe added p: critical / 2 and removed p: minor / 4 labels Nov 23, 2022

jhpalmieri added s: positive review and removed s: needs review labels Nov 23, 2022

dimpase added p: blocker / 1 and removed p: critical / 2 labels Dec 8, 2022

vbraun removed the s: positive review label Dec 11, 2022

vbraun closed this as completed in 7064a32 Dec 11, 2022

Update numpy to 1.23.5, scipy 1.9.3, networkx 2.8.8, meson_python 0.11.0 #34658

Update numpy to 1.23.5, scipy 1.9.3, networkx 2.8.8, meson_python 0.11.0 #34658

Comments

mkoeppe commented Oct 14, 2022

mkoeppe commented Oct 14, 2022

mkoeppe commented Oct 14, 2022

mkoeppe commented Oct 14, 2022

mkoeppe commented Oct 14, 2022

mkoeppe commented Oct 16, 2022

This comment has been minimized.

mkoeppe commented Oct 22, 2022

sagetrac-git mannequin commented Oct 22, 2022

sagetrac-git mannequin commented Oct 22, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

dimpase commented Oct 31, 2022

mkoeppe commented Nov 21, 2022

This comment has been minimized.

sagetrac-git mannequin commented Nov 21, 2022

sagetrac-git mannequin commented Nov 21, 2022

jhpalmieri commented Nov 22, 2022

jhpalmieri commented Nov 22, 2022

mkoeppe commented Nov 22, 2022

jhpalmieri commented Nov 22, 2022

mkoeppe commented Nov 22, 2022

jhpalmieri commented Nov 22, 2022

jhpalmieri commented Nov 22, 2022

mkoeppe commented Nov 22, 2022

mkoeppe commented Nov 23, 2022

mkoeppe commented Nov 23, 2022

jhpalmieri commented Nov 23, 2022

jhpalmieri commented Nov 23, 2022

mkoeppe commented Nov 23, 2022

dimpase commented Dec 8, 2022

vbraun commented Dec 11, 2022